These notes refer to version 1.1.5 of libUrl. This is the version that shipped with Revolution 2.7.1. Updaters for the most recent version of libUrl and other versions can be found here.
Note that versions of libUrl after and including 1.1 are not compatible with versions of Revolution earlier than 2.5. (More accurately, they are not compatible with engine versions earlier than 2.6.1.) The latest version of libUrl that is compatible with earlier engines is 1.0.16. If you use the updater for 1.1.5 (or later), it will install the version appropriate for the current engine. (The incompatibility is due to the support for secure https urls introduced in 1.1.1.)
The libUrl script library is used when the following Revolution commands and functions are called with HTTP, HTTPS or FTP urls:
load url <url> [with message <message>] unload url <url> get url <url> put <data> into url <url> post <data> to url <url> delete url <url> the cachedUrls urlStatus(<url>)
It also recognizes and implements the following properties:
the httpHeaders the httpProxy
In addition, the library contains a number of message and function handlers that can be called directly from your scripts. The names of these handlers are prefixed with "libUrl". They are described in the notes below.
Revolution commands and functions
load downloads the url and places it into a cache. The cached data will then be used in subsequent references to that url instead of it being downloaded again. Be sure to unload the url (using unload url) when you no longer need it.
The optional message parameter will cause that message to be called when the download completes (or when an error occurs). Two parameters are passed to the message: the url and its urlStatus. The message is sent to the same script where the load command originated.Example
on mouseUp put "http://www.xxxxxx.com/images/mykids.jpg" into tUrl load url tUrl with message "loadDone" end mouseUp on loadDone pUrl, pStatus if pStatus is "cached" then put url pUrl into image 1 of card 1 else answer "Download failed" end if unload url pUrl end loadDone
load is a "non-blocking" command. This means the url will load in the background while the script continues to run.
This allows you to monitor downloads, for example with a progress bar.
on mouseUp put "http://www.xxxxxx.com/images/mykids2.jpg" into tUrl load url tUrl showStatus end mouseUp on showStatus put "http://www.xxxxxx.com/images/mykids2.jpg" into tUrl put urlStatus(tUrl) into tStatus put tStatus ##show in message box if tStatus is not among the items of "cached,error,timeout" then send "showStatus" to me in 50 milliseconds end if end showStatus
See below for more information about urlStatus.
Note that because the load url command is executed in a script library and not by the engine directly, you must be careful not to include any "wait" commands in your script before the load has completed. "wait" will stop all scripts running, including libUrl. The following script will not work:
on mouseUp put "http://www.xxxxxx.com/images/mykids2.jpg" into tUrl load url tUrl wait until urlStatus(tUrl) is "cached" ##BAD end mouseUp
unload removes a previously loaded url from the cache and thus frees up memory. You should unload urls when you no longer need them.
get url downloads the url or, if the url is cached from a previous load command, retrieves it from the cache. If it completes successfully, the downloaded data will be in the variable it , and the result function will return empty. If an error occurs, the result function will return an error message.Example
on mouseUp put "http://www.xxxxxx.com/images/mykids.jpg" into tUrl get url tUrl if the result is empty then put it into image 1 else answer the result end if end mouseUp
Typically the error message will consist of the word "error" followed by a string containing more information about the error. Where appropriate, the error string will be the string returned by the http or ftp server including the server response code. For example:
error 404 File not found
Note that the variable it will not always be empty when an error occurs. Http servers typically return an "error page" for certain errors (401, 404, etc.) libUrl will try to download such data as well, and return it in the it variable. Because of this, you should always check the result function after a get url command. You can't assume that because it is not empty no error occurred.
get url is a "blocking" command. This means that the script lines following the get command will not continue until get url completes. However, because of the nature of libUrl, it doesn't block other scripts or user actions. If the get url command is in a button, the user could feasibly click the button again before the first mouseUp has completed. Or he or she may click another button that gets another url. In cases when a url is currently downloading from a get url command, subsequent get url commands will return the error "error Previous request has not completed." Because of this, you may want to disable interface elements while a get url command is completing, or take some other action that is appropriate to your application.Example
(similar script in a series of buttons)
global gUrlBlocking on mouseUp put "http://www.xxxxxx.com/images/mykids1.jpg" into tUrl if gUrlBlocking is true then beep else put true into gUrlBlocking get url tUrl put false into gUrlBlocking if the result is empty then put it into image 1 else answer the result end if end if end mouseUp
The same is true for the other "blocking" commands (post, put, delete). Only one url can be handled at a time with any of these commands.
The put command will save the data to the specified url on an ftp server. If it completes successfully, the result function will return empty. If an error occurs, the result function will return an error message. In the same way as for get url the error message will consist of the word "error" followed by a string containing more information about the error. Where appropriate, the error string will be the string returned by the ftp server including the server response code. For example:
error 530 Login incorrect.Example
on mouseUp put "ftp://dave:email@example.com/images/mykids.txt" into tUrl put field 1 into tData put tData into url tUrl if the result is empty then answer "Upload successful" else answer "Upload failed" & cr & the result end if end mouseUp
*See below about including user names and passwords for authorization
See get url for information about the "blocking" behavior of this command.
The post command is used to post data to an http server process such as a cgi script. If it completes successfully, the result function will return empty and the response from the server will be in the variable it. If an error occurs, the result function will return an error message.
See get url for information about the "blocking" behavior of this command.Example (for a typical form-to-email cgi)
on mouseUp put "http://www.xxxxxx.co.uk/cgi-bin/sendform" into tUrl put "subject=Post text" into tString put "&firstname.lastname@example.org" after tString put "&message=This is a test message from libUrl." after tString put "&page=http://www.xxxx.co.uk/submitted.html" after tString put crlf after tString post tString to url tUrl if the result is empty then put it into field 1 else answer the result end if end mouseUp
delete url will delete the file from an ftp server. If it completes successfully, the result function will return empty. If an error occurs, the result function will return an error message.
the cachedUrls function returns a list of currently cached urls, one per line. Note that it only includes those urls whose urlStatus is "cached". Urls whose urlStatus is "error", but for which there is cached data (e.g. as a result of a "file not found" error on an http server) will not be included in the list.
The urlStatus function returns the status of urls that have previously been referenced with a load url command. It will return one of the following:
queued contacted requested loading,x,y (where x = downloaded bytes and y = total bytes) timeout error cached
The function can be used to monitor files as they download (see load url above for an example), or to check the status of a url after the load has completed.
Note that if you upload data with libUrlFtpUpload (see below), instead of "loading" and "cached", urlStatus will return "uploading" and "uploaded" respectively.
Also note that "queued" is a new possible value. This is returned when a load url request has been made, and the url is placed on a queue while previous requests to the same host are completed.
The following library routines can be accessed directly from your scripts. More routines may be added from time to time. It's also possible that some of these routines may be incorporated into the Revolution language at some stage, although with different syntax.
This handler is basically the upload equivalent of the load url command and can be used to upload data to an ftp server. It is a "non-blocking" routine, so your script will continue to execute as the upload takes place in the background which allows the urlStatus to be monitored as the file is uploading. The optional message parameter lets you denote a message that will be called when the upload completes.
Note that urlStatus will return "uploading,x, y" and "uploaded" in replace of "loading,x,y" and "cached" when libUrlFtpLoad is used. Also note that urlStatus will continue to return "uploaded" for this url until you unload it with the unload url command. For this reason, you are advised to unload the url when you no longer need to know its status.Example
on mouseUp put "ftp://ftp.xxxxxx.com/literature/sadlife.txt" into tUrl put field 1 into tData libUrlFtpUpload tData, tUrl showStatus end mouseUp on showStatus put "ftp://ftp.xxxxxx.com/literature/sadlife.txt" into tUrl put urlStatus(tUrl) into tStatus put tStatus ##show in message box if tStatus is not among the items of "uploaded,error,timeout" then send "showStatus" to me in 50 milliseconds end if end showStatus
This function allows you to get further error data about a url whose urlStatus is "error". It will return the same string as the result of a get url command. Typically, this will be the response string returned by an http or ftp server. For example: "404 File not found"Example
put "ftp://ftp.xxxxxx.com/literature/sadlife.txt" into tUrl if urlStatus(tUrl) is "error" then get libUrlErrorData(tUrl) put "An error occurred" & cr & it into field "error" end if
This command allows you to switch between passive and active FTP data transfers. <mode> can be either "active" or "passive". As passive is the default, anything except "active" will cause the transfer to use passive mode. Any change will persist for the remainder of the session (i.e. until your application exits) or until you issue the command again. When your application starts, passive transfers will be used by default. So if you know you are going to be using active transfers, you should include this command before initiating any transfers.
If you don't know the difference between passive and active transfers, don't worry. Nine times out of ten, either will work fine. However, if you find that you are having trouble with ftp transfers, you might want to try switching to see if it makes a difference.Example
Warning: Don't use this command lightly. In fact, you should never have to use it at all. It closes all open sockets and clears all variables used by libUrl, including any cached data. It's basically the "panic button" when things go wrong, so it may be useful in development.
This command lets you set the number of seconds that a socket used for an ftp control connection remains open after a transaction completes. <seconds> must be an integer greater than 0. The default value is 15 seconds. If a new request to the same ftp user account is made while the socket is open, the connection is re-used and the timer reset. Note that libUrl makes no attempt to keep the connection open, so the server may close the connection first.Example
This command lets you set a field that will collect log data. The field parameter should evaluate to a valid field reference.Example
put the long id of field "log " into tField libUrlSetLogField tField
This function returns the current version of the libUrl library. There is no special significance to the numbers, but will be useful for reference when reporting any problems or difficulties.
This returns the headers of the most recent http request, including the request line.
This function returns the headers of the most recent reply from a server (remote host) to an http request.
Use this command to set your own custom headers for an http request. The <headers> argument must include the http request line.
Note that this command differs from the httpHeaders property. Header lines included in the httpHeaders property are added in addition to the default headers that are used for http requests. Headers in the httpHeaders property that have the same field title will replace the matching header in the default headers.
Headers set with libUrlSetCustomHttpHeaders will completely replace the request line and header fields used by default requests. This allows you to take complete control of http requests.
The custom headers set with libUrlSetCustomHttpHeaders are reset to empty after each request. So if you are using this command, you must use it before each request.
Lines in the headers argument should be separated by a Revolution/Metacard return character. libUrl will convert returns to crlf before sending the request.
libUrlFtpUploadFile uploads data directly from a file on the local drive to an ftp server. This is better for transferring very large files than using "put", which first loads to RAM all the data to be transferred.
The optional message parameter will cause that message to be called when the upload completes (or when an error occurs). Two parameters are passed with the message: the url and its urlStatus. The message should be in the same script as the libUrlFtpUploadFile command.Example
on mouseUp answer file "Select a file to upload" if it is not empty then put it into tSourcePath set the itemDel to "/" put last item of tSourcePath into tFileName put "ftp://ftp.xxxxxx.com/uploads/" & tFileName into tUrl libUrlFtpUploadFile tSourcePath, tUrl, "loadDone" end if end mouseUp on loadDone pUrl, pStatus if pStatus is not "uploaded" then answer "Upload failed" end if unload url pUrl end loadDone
libUrlDownloadToFile downloads data directly to a file on the local machine. This is better for transferring very large files than using "get" or "load", which first load all the data into RAM.
The optional message parameter will cause that message to be called when the upload completes (or when an error occurs). Two parameters are passed with the message: the url and its urlStatus. The message should be in the same script as the libUrlFtpUploadFile command.Example
on mouseUp ask file "Select a file to save to" if it is not empty then put it into tDestPath put "http://www.xxxxxx.com/stuff/greatstuff.jpg" into tUrl libUrlDownloadToFile tUrl, tDestPath, "loadDone" end if end mouseUp on loadDone pUrl, pStatus if pStatus is not "cached" then answer "Download failed" end if unload url pUrl end loadDone
libUrlSetStatusCallback allows you to set a callback message that will be sent during download and upload requests. The message sends status data of any current requests in a similar form to the urlStatus function. However, using a callback allows you to get status data even during blocking calls such as get url. This makes it easier to set up things such as progress bars.
To set the callback message, pass the name of the message and the long id of the object where the message handler resides.Example
on mouseUp libUrlSetStatusCallback "urlCallback", (the long id of stack "status") end mouseUp
libUrl will then send the message at the same time as it updates the urlStatus value. It passes two arguments with the message: the url and the current status. So, using the example above, you would set a handler in the stack script of stack "status".Example
on urlCallback pUrl, pStatus put pStatus into field "status" end urlCallback
The status argument will contain a string similar to that returned by urlStatus. The only difference that instead of "cached", "downloaded" is passed when a download has completed.
To turn off the callback feature, just send libUrlSetStatusCallback with an empty parameter.
libUrlSetFtpListCommand allows you to switch the ftp command that is sent when getting a directory listing. The choice is LIST or NLST. LIST returns the directory listing in the standard ftp style, which for most ftp servers is the Unix listing style. NLST returns only the file names in a return delimited list. The default is LIST, and passing anything other than NLST as an argument will cause the LIST command to be used.
Note that when using NLST, any sub-directory names are included in the returned data along with filenames.Example
libUrlFtpCommand allows you to send any ftp command to an ftp server. This is a powerful function that potentially provides full ftp support. However, to use it, you must be familiar with ftp commands and responses.
The <command string> parameter can be any FTP command string. For example, you would pass "PWD" to get the current working directory of the ftp server. The <host address> parameter is the address of the ftp server, for example "ftp.myserver.com" or "192.168.123.4". If the address doesn't include a port number, the standard ftp port (21) will be used. Unless you are connecting to an anonymous account, you should also pass the username and password for the ftp account.
The function returns the server response or an error message if an error occurred. A server response will always begin with a 3-digit code followed by any other relevant information. For example, a typical response to the PWD command would be: 257 "/" is current directoryExample
put "192.168.123.4" into tHost put "testfolder/file_1.cgi" into tFile put "dave" into tUser put "*****" into tPass##use a real password put "SITE CHMOD 755" && tFile into tCmd get libUrlFtpCommand(tCmd, tHost,tUser,tPass) answer it
libUrlFormData formats data in the standard format suitable for sending to form processing CGIs and other processes on a web server. The function accepts variable numbers of parameters and treats them as key-value pairs. The first parameter is the name of the first form part, the second the value of the first part, the third is the name of the second part, and so on.Example
put "John" into tName put "Hello" into tMessage get libUrlFormData("name", tName,"message", tMessage) post it to url "http://www.someserver.com/cgi-bin/form.pl"
In this case, the data posted to the url will look like this: name=John&message=HelloNote: The Content-Type header is set to "Content-Type: application/x-www-form-urlencoded" by default when using post. There is no need to set the httpHeaders unless you have previously set the Content-Type header to something else.
libUrlMultipartFormData formats data in the way described in rfc 1867. It was designed for posting files to web servers. The function can be called in a number of ways depending on the parameters passed. In all cases, the first parameter must be a variable which will be filled with the form data. The function will return empty if successful, or an error message if it fails (for example, if it couldn't open a file).
Note: When you need to supply a file as the value of a parameter, you must pass the file name preceded by the text "<file>".
Note: In all cases, the first line of the data returned in the form data variable is the Content-Type header that must be included in the httpHeaders of the url request. Lines 2 to the end of the data is the data that must be posted to the url.
The standard way to call the function is with pairs of name/value parameters.Example
put empty into tForm put "http://www.someserver.com/cgi-bin/form.cgi" into tUrl put "dave" into tName put "hello" into tMessage put "<file>" & "C:/myfile.txt" into tFile if libUrlMultipartFormData (tForm, "name", tName, "message", tMessage, "file", tFile) is not empty then answer it ##error else set the httpHeaders to line 1 of tForm post line 2 to -1 of tForm to url tUrl ## check the result, etc., here set the httpHeaders to empty end if
You can also pass in an array instead of pairs of parameters. This could be useful if there are many parts to a form. The array must be numerically indexed, and each element should contain the part name and part value, separated by a comma. So (using the above example):
put empty into tForm put "dave" into tName put "hello" into tMessage put "<file>" & "C:/myfile.txt" into tFile put "name," & tName into tArray put "message," & tMessage into tArray put "file," & tFile into tArray if libUrlMultipartFormData(tForm, tArray) is not empty then answer it ##error else set the httpHeaders to line 1 of tForm post line 2 to -1 of tForm to url <whatever> ## check the result, etc., here set the httpHeaders to empty end if
You can also call the function with no arguments except the form data. This will return an "empty" form. Basically, line 1 is the header, and line 2 is the final boundary mark of the form. It is of no use by itself, but it can be used with libUrlMultipartFormAddPart. See below for an example.
This lets you add parts to a multipart form one at a time. It also lets you optionally specify the mime type and transfer encoding for each part. This can be useful where the mime type or transfer encoding has to be specified.Example
put empty into tForm put "dave" into tName put "hello" into tMessage if libUrlMultipartFormData (tForm, "name", tName, "message", tMessage) is not empty then ##handle error and exit end if set the httpHeaders to line 1 of tForm delete line 1 of tForm put "<file>" & "C:/myfile.gif" into tFile put "image/gif" into tType put "binary" into tEnc if libUrlMultipartFormAddPart(tForm,"file", tFile, tType, tEnc) <> empty then ##handle error and exit else post tForm to url <whatever> set the httpHeaders to empty end if
libUrl_authcb_SetAuthToken <url>,<header string>,<token>,<use again>
These libUrl messages allow you to set a callback routine for handling authentication with http servers and proxies. Generally, these are enabling routines, and may require use of an external or other scripts when handling certain authentication schemes (for example NTLM or kerberos). libUrl provides support for handling the Basic authentication scheme only. Support for other schemes may be available from third parties.
Use libUrlSetAuthCallback to set a handler to be used for handling a particular authentication scheme. For example, to handle the Basic authentication scheme, you might do this:
libUrlSetAuthCallback "Basic", "authCallback"
where "authcallback" is the name of a handler. The hander should be in the same script as the handler that calls libUrlSetAuthCallback, or in that object's message path.
You can set callbacks for different Authentication methods. For example, the following two lines would set two handlers to be used for handling Basic and NTLM authentication schemes.
libUrlSetAuthCallback "Basic", "BasicAuthCallback" libUrlSetAuthCallback "NTLM", "NTLMAuthCallback"
The order of setting callbacks matters. Preference is given in the reverse order of the order that schemes are turned on. Where a server supports more than one scheme, the most recently turned on scheme that the server supports will be used.
(Note: To strictly follow the http rfc (2616), the schemes to be used should be turned on in order of weakest to strongest.)
The same callback handler is used for both Proxy and Server authentication. Your callback handler shouldn't have to make a distinction. libUrl will resolve which is being used.
You can stop using a callback by just calling libUrlSetAuthCallback again with an empty message. For example:
libUrlSetAuthCallback "Basic", ""
When a 407/401 response is detected by libUrl, the received headers are scanned for Proxy-Authenticate/WWW-Authenticate headers. The associated schemes are then matched against the callbacks already registered with libUrlSetAuthCallback. If a match is found, the callback message is sent to the target object (if it exists).
Two arguments are passed to the callback: the originally requested url and the related "authenticate" header received from the server. E.g.
Proxy-Authenticate: Basic realm="wonderland"
There are some rules for the callback handler. Basically it has to do 5 things:
- It has to create the appropriate token (credentials) for the server. (A libUrlBasicAuthToken() function that returns a suitable token for Basic schemes is now part of libUrl. An external will be required to provide the token/credentials for NTLM)
The token/credentials must begin with the name of the scheme being used. Essentially, it's the part that appears after the "Proxy-Authorization:" or "Authorization:" field title of the header.
- It has to "set" the token. Use libUrl_authcb_SetAuthToken to do this. You call it like this:
libUrl_authcb_SetAuthToken <url>, <HeaderString>, <Token>, <UseAgain><url> and <HeaderString> are exactly the same values that were passed into the callback handler. <token> is the token you have just obtained. <UseAgain> is true or false depending whether you want the token to be stored for re-use. With NTLM, you would set this to false, but with Basic or Digest you would typically set it to true. (When set to true, libUrl will automatically set Authorization headers on subsequent requests.)
- It has to resend the original request. This must be done with the libUrl_authcb_Resend handler. You just pass it the same url that was passed to the callback handler.
- It must return the result of the libUrl_authcb_Resend call. E.g
libUrl_authcb_Resend pUrl return the result
- It must have an "escape mechanism" if things go wrong. For example, a "Basic" scheme callback might put up a name/password dialog. The user can't remember the password and wants to Cancel. The only way to cleanly escape is to return empty.
Here is an example of a callback handler that handles Basic authentication.
on authCallback pUrl, pHeaderString local tRealm put "realm=(" & quote & ".+?" & quote & ")" into tRegEx get matchText(pHeaderString, tRegEx,tRealm) ask "Enter User Name for realm" && tRealm if it is empty then return empty put it into tName ask "Enter password for realm" && tRealm if it is empty then return empty put it into tPassword put libUrlBasicAuthToken(tName,tPassword) into tCredentials libUrl_authcb_SetAuthToken pUrl, pHeaderString, tCredentials, true libUrl_authcb_Resend pUrl return the result end authCallback
This function returns the Base64 encoded authorization token needed for the Basic authentication scheme. The following example:
will return this:
<limit> is a number of bytes.
When called, subsequent POST requests will add an "Expect: 100-continue" field to the http headers when the data to be posted exceeds the limit. The data won't be written to the socket until a 100 Continue response is received from the server.
This can be useful when posting large amounts of data and where there is a risk of the POST failing (because of failed authentication, or whatever). For example, if authentication is required, the data won't be written until after the authentication has cleared.
When set to true (the default), libUrl will respond to 301, 302, and 307 responses by trying to get the url listed in the Location header IF the original request was a GET (i.e not a post). It will respond to 303 responses by trying to GET the redirected url whatever the original request method. When set to false, no attempt is made to get the redirected url and the result will return "error" followed by the status code and message (e.g. error 302 found)
(This is different from previous behavior whereby libUrl always attempted to GET 301 and 302 redirects whatever the original request method, but didn't handle other responses.)
When set to true (the default), libUrl will use the "with verification" form of the "open secure socket" command which uses certificates set in the SSLCertificates property. When set to false, the "without verification" form is used which enables encrypted data transfer but without being certain who you're sending to or receiving from. Once set, the property is in use for all subsequent requests until it is set differently, so BE CAREFUL with this. (i.e. don't use it unless you know what you're doing)
If an http or ftp url requires authorization with a user name and password, these can be included in the url in the following way:
For anonymous ftp, no name or password is required. The library will add the "anonymous" user name and a dummy password automatically.
Note that the name and password included in this way with an http url will only work with the Basic authentication scheme, and only for server-based authentication. To handle other authentication schemes, or to handle Proxy Authentication, see libUrlSetAuthCallback.
If the user name or password contains non-alphanumeric characters (specifically the ":", "@", "/" , "." or "|" characters if these are allowed), then these should be urlEncoded before being put into the url.Example
put "jim" into tName put "email@example.com" into tPass put "ftp://" & tName & ":" & urlEncode(tPass) & "@ftp.xxx.com/title.txt" into tUrl get url tUrl
To use any of the above commands and functions, the libUrl script must be available. In the Revolution development environment, this is done automatically and there is no need to take any action. When building a standalone, however, you must transfer the library. In the Distribution Builder dialog, be sure that Internet Libraries is highlited under the Resources tab. There is no need to do anything more as all transferred libraries will be available when your application starts.
In the Metacard development environment, the libUrl script is automatically loaded when you use any of the engine calls that use libUrl (get, load, delete, etc.). However, it is not automatically loaded when using any of the commands and functions that start with "libUrl" unless one of the engine calls was made previously. In this case, you must "start using" libUrl before using a libUrl command or function. In all cases, before building a Metacard standalone, you should transfer the libUrl stack as a substack of your application's main stack. You can do this using the Resource Mover underneath the Tools menu. To start using libUrl, in your main stack you should insert the following line somewhere in your scripts so that it is called before any calls that make use of libUrl are made.
start using "libUrl"
April 11, 2006