2013-Aug-1 Meteor has a been successfully handed over to a new maintainer, and will continue to be supported in it's new home here.

Introduction to Meteor

Meteor is an open source HTTP server, designed to offer developers a simple means of integrating streaming data into web applications without the need for page refreshes.

It offers developers the freedom to think about web development in an entirely new way. It comprises the Meteor server and a standalone Javascript class which can be used in webpages to provide an abstraction layer for receiving data streams. Designed to be as simple and flexible as possible, Meteor offers a great solution for those wishing to add asynchronous functionality to their web projects.

Why Meteor?

A meteor is the visible event that occurs when a meteoroid or asteroid enters the earth's atmosphere and becomes brightly visible (thanks to Garth for correcting me on this!). Our Meteor an implementation of a technique called Comet for using the HTTP protocol for persistent streaming data connections. The term 'Comet' was coined by Alex Russell in his post 'Comet: Low Latency Data for the Browser'.

In fact, this technology has been around for ages, it's just difficult to implement well. Like AJAX before it, 'comet' only really became cool once someone gave it a name and showed the world in simple terms how it works. The main problem to be solved is one of scalability - most web servers are not designed to handle requests that take minutes or even hours to complete a response, and their threaded architectures collapse at the first sign of more than a few hundred simultaneously connected clients. As far as Apache is concerned, it's all about getting requests answered and closed as quickly as possible, and that's just not what's needed to make streaming work.

The guys behind the 'comet' concept are working on a new event-driven server architecture called Cometd, but in the meantime we needed a solution to a pressing need, so we built our own. A perl-based HTTP server written from the ground up to support high concurrency and longevity of connections, as well as memory-cached data to allow event-driven broadcasting of the same information to thousands of clients in near-realtime with minimal resource overhead and no disk access.

How it works

Meteor is two servers in one. It listens on one port for event controllers, and on another for channel subscribers. Event controllers are clients that connect on the control port and use Meteor's command protocol to inject events into named channels. Controllers can also issue commands to view the status of channels. Constructing a event controller is as easy as opening a socket and squirting a few simple text-based commands through it, trivial with most web programming platforms.

Subscribers are clients that connect on the subscriber port, and use the standard HTTP protocol to request a subscription to a particular channel or channels. A wide variety of querystring parameters can be included by the subscriber to indicate which interaction mode is desired.

Meteor then sends the events provided by the event controllers to the channel subscribers. All events are cached in memory so that the overhead required to send an event to a subscriber is minimal. In this way a few event controllers can do most of the hard work generating and formatting data, but without repetition, while Meteor handles the task of delivering the data to a large audience in near-real time.

Where to start?

If you're interested in knowing more about how Meteor works, read our intros to interaction modes and browser techniques. If you'd rather just make it work, try the installation instructions, or if you're really walking on the wild side, go straight to the download page. We're also eagerly inviting people to get involved and make improvements to Meteor, so if that's your bag, try the aptly-named how to join in page.

See also

There is a fair amount of activity in this area at the moment. For more information elsewhere on the net we recommend reading:


Interaction modes

Meteor is designed to support multiple methods of pushing (or appearing to push) data over HTTP. Since HTTP is a protocol that is designed to 'pull', streaming doesn't come naturally and some 'creative' use of the protocol is required. Some of these methods work in some circumstances, and not in others, while each has its own pros and cons.

Streaming

Network traffic diagram demonstrating streaming

To stream data, a client initiates a request, the server's response begins immediately, and continues indefinitely until the client closes the connection. This would seem to be the ideal method of interaction - events can be pushed out as they happen on a pre-established connection, the resources spent opening and closing sockets are minimised, and since the connection has already been negoitiated, no additional headers or wrapper content are required so the use of bandwidth is also very efficient.

The problems with streaming are evident when the connection is routed to, or via, hosts that are not willing to play ball with this style of interaction. First, web browsers often wait for the entire response to complete before processing it. Second, proxy servers often wait for a response to complete before passing it on to the client, so streaming connections may never get received.

On the server side there are issues as well. By having no timeout on connections, it's possible for lots of zombie connections to slowly build up, particularly if the client has failed to disconnect but is no longer listening to the response. This is particularly the case with proxy caches that try to finish downloading files even when their client has disconencted, so that it's cached for the next request. And even assuming no zombie connections, there is still the issue of conccurrency - thousands of open sockets with not very much happening on most of them at any given moment.

Overall, streaming connections are still a very compelling choice where the browser can cope with reading a partially downloaded response, and there are no buffers in the way.

Short polling

Network traffic diagram demonstrating short polling

Short polling cannot pretend to even emulate pushing data to the client, but it is one step up from refreshing an entire page. Using AJAX, the client connects to the server and requests new events. The server sends an immediate response containing the events that have occured since the client last requested an update, and closes the connection. The client waits a predetermined interval and initiates another request.

Using this method, client and server spent the majority of their time not connected to each other, with the regular periodic polling connections being answered and closed by the server within a few milliseconds. This reduces the number of concurrent connections required of the server, but if there are lots of subscribers and the updates being requested by each one require an expensive but repetitive backend operation (like a database query), the server can rapidly become overloaded doing unnecessary tasks over and over again. Meteor addresses this by storing the events in memory and answering polling requests directly from the memory cache, avoiding repetitive database or disk access. This means that even though Meteor is designed to deal with long-lived connections, it is still worth using for polling as well.

Short polling is the most reliable way of updating data that will generally survive most browser and connection setups, but it's hardly ideal as it's not particularly scalable and the client is by no means receiving updates as they happen.

Long polling

Network traffic diagram demonstrating long polling

Sounds like it's time for a happy middle ground between streaming and short polling, doesn't it? Well, long polling may well fit the bill. The client initiates a request, and if the server has events pending, it sends them and closes the connection, much like short polling. But if the server does not have any events waiting, it holds the connection open until an event occurs, at which point it send the event and closes the connection. The client can then initiate a new connection immediately, since all the waiting is done at the server end.

The obvious benefit to this is that the client can treat the interaction as a simple 'request-response' and wait for it to complete before processing it rather than having to sniff the response as it loads. It works through proxies and is more resilient to connections dropping. And since the server is actively closing connections all the time, the chance of long-lived zombie connections is much reduced. On the negative side, there is a need to create and close many more connections than for a stream, one for each event that is sent (but far fewer than for short polling, because the server no longer has the thankless task of responding to millions of repeated requests with "no updated events available"). Also events may be slightly delayed if they occur in rapid succession while the client is reinitiating a new connection after the last update.


Browser techniques

This is a description of the various methods Meteor uses in the browser to interpret responses from the Meteor server. As usual it's a hotch-potch of browser incompatibilties, but thankfully there's something that works for pretty much every combination.

Any requirement to load additional data asynchronously into a page once the page has loaded requires making a new HTTP connection. The common pitfalls of such an action are:

  1. 'Click' navigation noises in Internet Explorer
  2. An unwanted entry in the browser history preventing use of the 'Back' button
  3. Inability to access the data until it has finished loading
  4. 'Loading'-type effects, like progress bars and spinners
  5. Cross site scripting security restrictions

And the methods, or transports, available are:

  • IFRAME ("Forever frame"): Loading a page in an IFRAME that incrementally receives commands wrapped in <script> tags, which the browser evaluates as they are received.
  • XMLHTTPRequest Interactive: An AJAX request is made and the browser fires events every time data is received, even if the response is not yet complete. The entire response (so far) can then be read from the responseText property.
  • XMLHTTPRequest (polling only): An AJAX request is made and the browser fires events every time data is received, even if the response is not yet complete. The entire response (so far) can then be read from the responseText property.
  • Dynamic SCRIPT loading ("CometP", polling only): By adding a SCRIPT tag to the HEAD of the document and setting its src attribute, new script can be loaded and evaluated by the browser.

The techniques described here are those employed by Meteor's Javascript client to achieve the best result in each browser without any of the issues documented at the top of this page, and use a combination of several of the transports listed above. Details of the browser capability tests that led to these techniques being developed are documented in the cross site scripting matrix.

Forever Frame via HTMLFile ActiveX Object

Useful for: Streaming in MSIE

The IFRAME method normally produces navigation clicks and 'loading' actions in IE, so to get around this we can take advantage of a nifty IE-only solution - the HTMLFile ActiveX object, which must have been dreamt up by someone who has never heard of standards. Not that we're complaining. This neat solution essentially creates an in-memory page, in which you can, conveniently, have an IFRAME that's dynamically loading something. Here's how it works:

var transferDoc = new ActiveXObject("htmlfile");
transferDoc.open();
transferDoc.write("<html><script>");
transferDoc.write("document.domain=\""+(document.domain)+"\";");
transferDoc.write("</"+"script></html>");
transferDoc.parentWindow.Meteor = Meteor;
transferDoc.close();
var ifrDiv = transferDoc.createElement("div");
transferDoc.appendChild(ifrDiv);
ifrDiv.innerHTML = "<iframe src=\""+url+"\"></iframe>";

The page loading in the hidden iframe is incrementally loading and executing a set of function calls, and can pass its instructions up to the parent page. Notice that a reference to the Meteor class is passed into the HTMLFile 'page'.

The IFRAME code itself is sent to the browser by Meteor server, which, depending on how it's configured, will likely send something like this:

<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<script>
window.onError = null;
var domainparts = document.domain.split(".");
document.domain = domainparts[domainparts.length-2] + "." + domainparts[domainparts.length-1];
parent.Meteor.register();
</script>
</head>
<body onload="r()">
<script>p(-1,"{hf:'ml_ping'}");</script>
<script>p(-1,"{hf:'ml_ping'}");</script>
....

Every time a <script> chunk is received, the javascript is evaluated, exeuting the local p() function, which is actually a link to the process function in the parent frame (a link established by the parent.Meteor.register() function call).

XMLHTTPRequest Interactive State

Useful for: Streaming in Firefox and Safari

The XMLHTTPRequest in these browsers supports the interactive readyState, and fires events every time a lump of data is received.

To ensure we can make an XHR to the meteor server on a subdomain, we need to issue the XHR from the same host, so we still load an IFRAME, but the page loaded into the frame is simply a static page of Javascript that holds and manipulates the XMLHTTPRequest object that will connect to Meteor for the data stream. This page is served by Meteor so it satisfies the strict same origin policy for XHR, but can also talk to the parent frame thanks to a looser same origin policy for frames (if you're going cross eyed at this point, refer to the cross site scripting matrix).

streamreq = newXmlHttp();
byteoffset = 0;
var url = parent.Meteor.getSubsUrl();
streamreq.open("GET", url, true);
streamreq.onreadystatechange = function() {
	if (typeof streamreq == "undefined") return;
	if (streamreq.readyState == 3) {
		var buffer = streamreq.responseText;
		var newdata = buffer.substring(byteoffset);
		byteoffset = buffer.length;
		while (1) {
			var x = newdata.indexOf("<s"+"cript>p(");
			if (x != -1) {
				y = newdata.indexOf("</"+"script>", x);
				if (y != -1) {
					eval(newdata.substring((x+8),y));
					newdata = newdata.substring(y+9);
				} else {

					// Last message is incomplete.  Ignore it and it will be fetched again
					break;
				}
			} else {

				// No more messages
				break;
			}
		}
		byteoffset = buffer.length-newdata.length;
	} else if (streamreq.readyState == 4) {
		delete streamreq;
		if (typeof(r)=="function") {
			r();
		}
	}
}
streamreq.send(null);

The magic happens between lines 8 and 29. First we get the latest contents of the receive buffer, which will contain a page similar to the one shown in the forever frame techniques. However, rather than it being loaded and processed incrementally as a page in its own right, we're now going to just treat it as pure data. So we ignore all the header HTML and functions, and start hunting for the <script>p( sections. Having found some (and there might be several new ones on every interactive event), we eval() them one at a time.

Normal XMLHTTPRequest

Useful for: Polling in all browsers

You can implement polling using an XMLHTTPRequest in all browsers, as you don't need to have access to the data stream until the response is complete. To circumvent cross site scripting restrictions (the AJAX target being on a different host), we'll operate the polling client from an IFRAME whose source is loaded using Meteor's static page serving feature. Here's the relevant bit of code from the IFRAME:

try {
	document.domain = thisdomain;
} catch (e) {}
pollreq = newXmlHttp();
pollreq.open("GET", url, true);
pollreq.onreadystatechange = function() {
	if (pollreq.readyState == 4) {
		document.domain=topdomain;
		newdata = pollreq.responseText;
		while (1) {
			var x = newdata.indexOf("<s"+"cript>p(");
			if (x != -1) {
				y = newdata.indexOf("</"+"script>", x);
				if (y != -1) {
					eval(newdata.substring((x+8),y));
					newdata = newdata.substring(y+9);
				} else {
					break;
				}
			} else {
				break;
			}
		}
		pollreq = false;
	}
};
pollreq.send(null);

The code is run repeatedly at regular intervals using a standard setTimeOut technique. Before the start of every request, document.domain is set to the iframe's domain (ie. Meteor's hostname), eg. push.example.com; then, before dealing with the data retrieved from the response, the domain is set back to the top level domain that the IFRAME and the parent page share, ie. example.com. This is because in Internet Explorer, XMLHTTPRequests can only access URLs on the exact host from which the containing page was requested, while inter-frame data exchange can only happen when both pages share the same document.domain. So it needs to be changed every time we make a request. The setting of the document.domain parameter is in a try...catch enclosure because it will fail in Firefox, which only allows document.domain to be shortened. However, this doesn't matter, because Firefox will in any case allow XMLHTTPRequests despite document.domain being set to a host that does not match the target of the request.

To avoid back-button issues, we'll load this polling iframe into an HTMLFile in IE, and a double IFRAME in Firefox and Safari.

Simple IFRAME Forever Frame

Useful for: Streaming in Opera

The IFRAME method is the best Meteor can do in Opera, which supports neither the XHR Interactive readystate, nor the HTMLFile activeX object. So this method continues to suffer from loading effects. However, Opera does support "server-sent events", which we hope to add to Meteor soon.


Cross site scripting support

Because Meteor and your normal web server have to live side by side, we had to find ways of loading information from Meteor in pages served by Apache, after the page has finished loading. This page examines all the ways this can be done and documents which ones work for which browsers.

There are three main mechanisms that can be used to load data asyncronously once a page is loaded:

  • XMLHTTPRequest (XHR)
  • IFRAME
  • SCRIPT

All of these work well enough when retrieving data from the same server that served the page (at least in all the browsers under test below), but the problems arise when you want to retireve data from a different host or port, because you run into security restrictions implemented in the browser.

Some other comet implementations include a proxy engine that allows the comet server to handle all the requests directly, and pass non-comet requests through to a hidden web server, but Meteor does not support that style of behaviour, and there are some very good reasons to avoid doing it in this way.

The behaviours

All these results come from observations using tests which are no longer available online, unfortunatly.

T'port Configuration IE 6/7 FF 2 Op 9 Saf 2 Saf 3 iPhone
XHR one-part, same-origin Yes Yes Yes Yes Yes Yes
one-part, same host, different port Yes No No No No No
one-part, parent host, same port No No Yes2 No No No
one-part, parent host, different port No No No No No No
incremental, same-origin No8 Yes No9 No3 No3 Yes
incremental, same-origin, 1K prepended 'noise' No8 Yes No9 Yes Yes Yes
incremental, same host, different port No8 No No9 No No No
incremental, same host, different port, 1K prepended 'noise' No8 No No9 No No No
incremental, parent host, same port No No No9 No No No
incremental, parent host, same port, 1K prepended 'noise' No No No9 No No No
incremental, parent host, different port No No No No No No
incremental, parent host, different port, 1K prepended 'noise' No No No No No No
T'port Configuration IE 6/7 FF 2 Op 9 Saf 2 Saf 3 iPhone
IFRAME one-part, same-origin Yes1 Yes1 Yes1 Yes1 Yes Yes
one-part, same host, different port Yes1 Yes1 No No Yes1 No
one-part, parent host, same port Yes1 Yes1 Yes1 Yes1 Yes1 Yes1
one-part, parent host, different port Yes1 Yes1 No No Yes1 No
incremental, same-origin No3 Yes1 Yes1 No3 No3 Yes
incremental, same-origin, 1K prepended 'noise' Yes1 Yes1 Yes1 Yes Yes Yes
incremental, same host, different port No3 Yes1 No No No3 No
incremental, same host, different port, 1K prepended 'noise' Yes1 Yes1 No No Yes1 No
incremental, parent host, same port No3 Yes1 Yes1 No3 No3 Yes
incremental, parent host, same port, 1K prepended 'noise' Yes1 Yes1 Yes1 Yes1 Yes1 Yes
incremental, parent host, different port No3 Yes1 No No No3 No
incremental, parent host, different port, 1K prepended 'noise' Yes1 Yes1 No No Yes1 No
T'port Configuration IE 6/7 FF 2 Op 9 Saf 2 Saf 3 iPhone
SCRIPT one-part, same-origin Yes Yes Yes Yes Yes Yes
one-part, same host, different port Yes Yes Yes No Yes No
one-part, parent host, same port Yes Yes Yes Yes Yes Yes
one-part, parent host, different port Yes Yes Yes No Yes No
any incremental-loading configuration No No No No No No
Shorten document.domain Yes Yes Yes Yes Yes Yes
Lengthen document.domain to original length once shortened Yes No No No7 No7 No7
T'port Configuration IE 6/7 FF 2 Op 9 Saf 2 Saf 3 iPhone

Table notes

  1. IFRAME can only be accessed if its document.domain matches the parent frame's document.domain
  2. XHR can only be made if document.domain is a match for or a parent of the target host. This will always be the case for XHRs to the originating host, but document.domain must be shortened to allow XHR to the parent domain.
  3. Safari and IE have a buffer which must fill up before any response is parsed. Data received before the buffer is full will not be rendered or interpreted and will not fire the Interactive state of an XHR until the buffer is full or the connection is closed. The obvious solution to this is to send a blob of 'noise' at the start of any response that needs to be parsed incrementally. The size of the buffer that needs to be filled is 512 bytes in Internet Explorer, and between 256 bytes and 1K in Safari depending on the Content-type of the response (a non-standard content-type equals a 256b buffer, while a text/html content type may be as much as 1K)
  4. SCRIPT loading is a blocking action in Opera, non-blocking in other browsers.
  5. In Safari, HTTP URLs that do not explicitly define a port number (and therefore default to port 80 in all browsers) are considered to have a different port to those URLs that do explicity define port 80, even though both URLs actually use the same port. The test suite therefore defines 'same port' as 'no port specified', rather than explicitly setting :80.
  6. Despite setting cache-busting headers, you must give every script you load using the SCRIPT transport a unique URL, else the browser may simply pull the existing script out of memory and run it again.
  7. Fails silently, does not throw an error
  8. In Internet Explorer, the first incremental response to XHRs does fire a readyState 3 'interactive' event, but neither the body nor the headers of the response are available until the response is complete, making these events essentially useless. So for the purposes of incrementally downloading data into the browser, the incremental XHR is considered non-functional in IE.
  9. Although the responseText property of the XHR is available while the response is still being received, Opera only fires the readyState 3 'interactive' event once, so subsequent packets received will not be processed.

Conclusions

The way Meteor is designed to work is compatible with all of these browsers, in that a subdomain frame can talk to a parent domain frame, and all actual server conenctions can be made to same-origin from the subdomain frame. Separating the comet server from the web server using ports won't work in Safari 2 or Opera.


Server documentation

How to configure / run / tweak your Meteor server to do pretty much anything. If you just want to do basic streaming, try installing first :-)

Starting the Meteor daemon

To start Meteor, use the cd command to change to the directory the meteord daemon is in, then start with:

./meteord -d

This will run Meteor in debug mode, in which it will print out debug information to the console. If the -d flag is omitted, Meteor will start itself as a daemon. See below for a full list of possible configuration parameters.

To properly install Meteor copy the Meteor directory to one of the paths at which perl5 finds it’s modules, and copy the meteord program to any desired location from which it can be started.

Server configuration parameters

This section lists all the configuration parameters that can be set when Meteor is started. Meteor will always determine a value for every parameter, with the following precedence:

  1. Command line
  2. Config file
  3. Default

To set config parameters on the command line, start Meteor as follows:

meteord [-parameter [value] [-parameter [value]...]]

On the command line, the parameter names are not case sensitive, and the shortest abbreviation that uniquely identifies a parameter can be used. For example the debug parameter can be shortened to d as no other parameters start with that letter. If a parameter name is given with no value a value of 1 will be assumed (and will override any conflicting setting in the config file). If you wish to set a value that starts with ‘-‘, prefix it with an additional ‘-‘. Finally, in addition to the parameters listed below Meteor supports the optional command-line only parameter -help (or -h), which outputs the following list, as well as a sample configuration file.

The configuration file can also list any of the parameters, one per line followed by whitespace and the parameter value. A leading space or tab in a parameter can be written as \s or \t respectively. Carriage returns and linefeeds must be written as \r and \n respectively. Backslashes must be doubled up. No abbreviations are allowed in the configuration file. Any lines beginning with a hash (#) character are comments and will be ignored, as are empty lines (only containing whitespace).

Some configuration parameters can be overridden for a specific connection mode. In the sample configuration file these overrides are used to define the differences between the transports that are used in various browsers. For example, streaming in Firefox uses the interactive state of an XMLHTTPRequest, while Internet Explorer loads the stream into an iframe as if it's an HTML file. As a result the messages need to be packaged differently, and that's where some of the configuration parameters can be overridden to define these differences.

To define a section of the config file that relates to a particular transport mode, enclose the mode name in square brackets (this format should be familiar to anyone that has edited Windows ini files):

[flash]
HeaderTemplate ~channelinfo~
Persist 1
MessageTemplate ~id~/~channel~/~text~\0
PingMessage .\0
SubscriberShutdownMsg x\0
ChannelInfoTemplate CH/~name~/~lastMsgID~\0

The following configuration parameters can be overridden for a specific transport mode:

  • PingMessage
  • MaxTime
  • MaxMessages
  • HeaderTemplate
  • SubscriberShutdownMessage
  • Persist
  • MessageTemplate
  • ChannelInfoTemplate

Note that the ConfigFileLocation parameter can only be set from the command line!

ChannelInfoTemplate
Template for each line of channel information. Channel information can be included in a HeaderTemplate using ~channelinfo~, and if included, one line of channel information will be printed for each channel to which the request is subscribed. The placeholders ~name~, ~lastMsgID~, ~messagecount~ and ~subscribercount~ can be used in the ChannelInfoTemplate and will be replaced with appropriate values. The Meteor JS client uses channel information to find out the latest message index on each channel. Format: String, Default: <script>ch("~name~", ~lastMsgID~);</script>
ConfigFileLocation
Configuration file location on disk (if any). Format: String, Default: /etc/meteord.conf
ControllerIP
IP address for controller server (blank = all local addresses). Format: Dotted decimal, Default: blank

ControllerPort
Port number for controller connections. Format: Integer, Default: 4671

ControllerShutdownMsg
Controller Shutdown message, sent to all connected event controllers when Meteor shuts down (leave empty for no message). Format: String, Default: blank

Debug
Debug Flag, when set daemon will run in foreground and emit debug messages. Options: [1|0], Default: 0
DirectoryIndex
Name of index file to serve when a directory is requested from the static file web server. Format: String, Default: index.html
HeaderTemplate
Header template, content to send to subscriber clients immediately upon successful connection, before any messages. ~server~, ~servertime~, ~status~ and ~channelinfo~ are replaced by the appropriate values. See ChannelInfoTemplate for more details about channel information. Format: String, Default:
HTTP/1.1 ~status~\r\nServer: ~server~\r\nContent-Type: text/html; charset=utf-8\r\nPragma: no-cache\r\nCache-Control: no-cache, no-store, must-revalidate\r\nExpires: Thu, 1 Jan 1970 00:00:00 GMT\r\n\r\n

Help
Print a help message to standard output and exit. Options: [1|0], Default: 0
LogTimeFormat
Format in which to print timestamps on log output. Choose from unix timestamp (unix) or RFC2822 formatted date (human). Options: [unix|human], Default: human
MaxMessageAge
Maximum age of a message in seconds, after which the message will be purged from the memory cache. Format: Integer, Default: 7200

MaxMessages
Maximum number of messages to send to a subscriber before forcing their connection to close. Use 0 to disable. Format: Integer, Default: 0

MaxMessagesPerChannel
Maximum number of stored messages per channel. When this limit is reached, the oldest messages will be removed as new ones are added. Format: Integer, Default: 250

MaxTime
Maximum duration in seconds for a subscriber connection to exist before forcing it to close. Note that the server checks for expired connections in 60 second intervals, so small changes to this value will not have much of an effect. Use 0 to disable. Format: Integer, Default: 0

MessageTemplate
Message template, content to send for each event: ~text~, ~id~, ~channel~ and ~timestamp~ will be replaced by the appropriate values. Format: String, Default: <script>p(~id~,"~channel~","~text~");</script>\r\n

Persist
Whether connections should persist or be closed as soon as the first response has been made. Options: [0|1], Default: 0

PingInterval
Interval at which PingMessage is sent to all persistent subscriber connections. Must be at least 3 if set higher than zero. Set to zero to disable. Format: Integer, Default: 0

PingMessage
Message to be sent to all persistent subscriber connections every PingInterval seconds. Format: String, Default:
<script>p(-1,"");\r\n</script>\r\n

SubscriberIP
IP address for subscriber server (blank = all local addresses). Format: Dotted decimal, Default: blank

SubscriberPort
Port number for subscriber connections. Format: Integer, Default: 4670

SubscriberShutdownMsg
Subscriber Shutdown message, sent to all connected subscribers when Meteor shuts down or the same client reconnects from a different source (leave empty for no message). Format: String, Default:
<script>eof();\r\n</script>\r\n

SubscriberDocumentRoot
An absolute filesystem path, to be used as the document root for Meteor's static file web server. If left empty, no documents will be served. Format: String, Default: /usr/local/meteor/public_html
SubscriberDynamicPageAddress
Since Meteor is capable of serving static pages from a document root as well as streaming events to subscribers, this paramter is used to specify the URI at which the event server can be reached. If set to the root, Meteor will lose the ability to serve static pages. Format: String, Default: /push

SyslogFacility
The syslog facility to use. Format: String, Default: daemon

UDPIP
Experimental UDP server support - set this to the IP on which to listen for controller commands over UDP. Leave blank for all local addresses. Format: String, Default: ''

UDPPort
Experimental UDP server support - set this to the port on which to listen for UDP connections. Set to zero to disable UDP server. Format: String, Default: 0

Subscriber request format

Subscriber clients connect to Meteor and issue standard HTTP-formatted requests. These can be in two flavours: static page requests and data channel subscriptions. The static page web server is a very basic feature of Meteor designed simply to allow a few pages to be served from the same host/port as the data stream, which prevents cross site scripting restrictions from causing a problem. A request for a static page should be in the form:

GET /path/to/pagename.html HTTP/1.1\r\n\r\n

The GET request can include querystring parameters or anchors if desired, but these will be ignored by Meteor (and are generally used for cache-busting by the client). Additional headers such as cookies and user agent strings can also be sent in the request, and will also be ignored.

Static page serving is enabled if the server config parameter SubscriberDocumentRoot is set to a valid filesystem path. Files will only be served if each component of the file path contains only the characters A-Z a-z 0-9 -_. and does not start with a dot.

Once a file has beens served once, it is read into memory along with the Meteor message queue. Further requests for the same file are handled from the memory cache. That means the files should be small.

If the requested path points to a directory, the file indicated by the DirectoryIndex variable is served if it exists in that directory.

Requests for subscriptions to channels should be in the form:

GET /push/{hostid}/{streamtype}/{channeldef1} [/{channeldef2}[/{channeldef3}[...]]] HTTP/1.1\r\n\r\n

In this case the SubscriberDynamicPageAddress server configuration parameter is assumed to be push. The remaining elements of the path are:

hostid
Gives the server a unique reference for the client. The server will use this to ensure that it is only serving one response to each client and reduces the liklihood of zombie connections. If a client opens a connection of any kind, specifying an id, then any persistent connections with the same id will be closed by Meteor. Mandatory. Format: String
streamtype
The type of connection to establish. Options are iframe, xhrinteractive, longpoll, smartpoll, simplepoll Mandatory. Format: String
channeldefs
The client may subscribe to multiple channels per request, and should include a channel definition for each one. This is made up of the channel name, and then optionally, a restart command. Restart commands consist of a dot and the letter r followed by a message index number to restart at that message, the letter b followed by an offset to go back that number of messages, or the letter h to retrieve the full history of the channel. If no restart command is provided, Meteor sends only events that occur after the request is received. Valid channel def examples are: demo.r456, chatroom.b5, channel1, channel2.h. Mandatory, repeatable

The combination of the parameters above can be used to request all the possible interaction modes, eg:

# Stream via iframe on channel1 and channel2
GET /push/12345/iframe/channel1/channel2 HTTP/1.1

# Poll channel1 for last message and channel2 for all messages from index 8845
GET /push/12345/simplepoll/channel1.b1/channel2.r8845 HTTP/1.1

# Long poll channel 1 starting at index 8845
GET /push/12345/longpoll/channel1.r8845 HTTP/1.1

Note that when short polling (using streamtype 'simplepoll' or 'smartpoll'), if the client does not request a backtrack (.b), full history (.h) or a resume (.r), then no messages will be returned because the connection will not be open for long enough for any events to occur. However, this may still be desirable. If the client does not know where to resume from and does not want to receive any existing messages, it may make a simple polling request for the channel with no suffix, and Meteor will respond with a channel information summary, enabling the client to know where to resume from on it's subsequent request.

This may sound slightly confusing. To step though it:

  1. Client wants to POLL, rather than stream (probably because it has already tried streaming and it didn't make it through a proxy or firewall). However, client is yet to receive any data and does not know what the index of the latest message is
  2. Client issues GET /push/1/simplepoll/testchannel
  3. Meteor replies ch('testchannel', 44324)
  4. Client now knows the most recent message index and can request GET /push/1/simplepoll/testchannel.r44325
  5. Meteor replies with another channel info summary and any messages with an ID greater than or equal to 44325

Event controller command protocol

Event controllers connect to Meteor on the control port, and talk in Meteor's basic command protocol. Connections are not authenticated so you should ensure that the control port is protected by your firewall. The following commands are understood:

ADDMESSAGE channelName messagetext

Create a new message (and a channel if the specified channel does not exist yet).

COUNTSUBSCRIBERS channelName

Returns the string OK followed by a space and the number of subscribers on the specified channel.

LISTCHANNELS

Lists all known channels, the number of messages stored on each one and the number of current subscribers. The response has this format:

OK
channel1(1/0)
channel2(3/0)
--EOT--

The first line indicates the command was understood. Then all existing channels are listed one channel per line. The channel name is followed by the number of messages and the number of subscribers, separated by a slash and enclosed in brackets. The string --EOT-- on a line by itself ends the listing.

SHOWSTATS

Outputs activity statistics:

OK
uptime: 117
channel_count: 2
subscriber_connections_accepted: 3
controller_connections_accepted: 3
total_inbound_bytes: 1851
total_outbound_bytes: 14978
current_controllers: 1
unique_messages: 10
total_requests: 6
documents_served: 2
current_subscribers: 1
--EOT--

These stats are relative to the time the Meteor process was started and are updated as events occur, so controllers wishing to analyse time-based stats should sample the data periodically and construct a rolling average. If Meteor stats are of interest to subscribers, eg to let them know how many other subscribers are connected, a controller could periodically fire these stats back into Meteor as a channel message.

QUIT

Returns the string OK and closes the controller connection.

Signals

Meteor understands the HUP, TERM, USR1 and USR2 signals:

SIGHUP
When Meteor receives a HUP signal, it will re-read the configuration file, then empty the document cache as the document root may have changed.
SIGUSR1
When Meteor receives a USR1 signal, it will empty all channel caches.
SIGUSR2
When Meteor receives a USR2 signal, it will empty the document cache.
SIGTERM
When Meteor receives a TERM signal, it will perform a clean shutdown as follows:
  1. Close the Subscriber server
  2. Close the Controller server
  3. Delete the /var/run/meteord.pid file
  4. Close all connections after sending any configured shutdown messages
  5. Exit

During a TERM procedure, Meteor will attempt to send the shutdown messages for a maximum of 120 seconds. If some clients can not be contacted during that time, it will exit with an exit value of 1. Otherwise it will exit with a value of 0.


JS Client documentation

How to use the Meteor Javascript client to connect to a Meteor server

Dependencies

Before you can use Meteor in your page, you need to include the object library in the header of your HTML files. Meteor server hosts the Javascript client on its own host, and serves it directly, so there is no need to include any Meteor files in your web application.

<script type="text/javascript" src="http://data.example.com/meteor.js"></script>

Meteor presents itself to your application as a javascript object, so there is no need to instantiate it.

Properties

channelcount
Number of channels currently subscribed. Format: Integer, cannot set, read anytime
debugmode
Puts Meteor client in debug mode. If the console object exists, it writes to console, otherwise it looks for a DOM element with the ID 'meteorlogoutput' and appends to that. Format: Boolean, set anytime, read anytime
scheme
The protocol scheme, either http or https for communicating with Meteor server. Format: String, Default: http, set before connect(), read anytime
host
The host on which the Meteor server is listening for connections. Format: String, set before connect(), read anytime
port
The port on which the Meteor server is listening for connections. Format: String, Default: 80, set before connect(), read anytime
lastpingtime
Millisecond timestamp (ie. value of Date.getTime()) of the last ping received from the server. Format: integer, cannot set, read anytime
hostid
Unique identifier for this client, to prevent multiple Meteor connections being made from the same client to the same server (which would saturate the 2-connection limit in most browsers). It is normally sensible to generate this within the web application server-side, and to store it in the user's session. Format: integer, set before connect(), read anytime
minpollfreq
Absolute minimum number of milliseconds between polling requests, when Meteor is in polling mode and smart polling is enabled. Format: Integer, Default: 2000, set anytime, read anytime
maxpollfreq
Absolute maximum number of milliseconds between polling requests, when Meteor is in polling mode and smart polling is enabled. Format: Integer, Default: 60000, set anytime, read anytime
mode
Set to stream, simplepoll, smartpoll or longpoll. Meteor will auto-select the appropriate streaming transport for streaming connections, but you can override this by setting mode to iframe or xhrinteractive. If started in streaming mode, Meteor may fall back to smart polling mode if the stream is interrupted. Options: [stream|simplepoll|smartpoll|longpoll|xhrinteractive|iframe], Default: stream, set before connect(), read anytime
pingtimeout
Number of milliseconds to allow between pings before reverting a streaming connection to polling mode. Should be set to at least three times the ping interval defined in the Meteor server config. Format: integer, Default: 0, set anytime, read anytime
pollfreq
Number of milliseconds between polling requests, when Meteor is in polling mode. Will gradually drift from this starting value if in smartpoll mode. Format: Integer, Default: 2000, set anytime, read anytime
polltimeout
Number of milliseconds to allow for a polling request to return a response from Meteor server. In the case of long polling connections, this may be a long time so the value should be set high, whereas short polling connections will want a short timeout. Format: integer, Default: 30000, set anytime, read anytime
status
Current status of the client. Possible values are: 0 = Uninitialised, 1 = Loading stream, 2 = Loading controller frame, 3 = Controller frame timeout, 4 = Controller frame loaded and ready, 5 = Receiving data Format: integer, Default: 0, cannot set, read anytime

Methods

joinChannel(channelname, backtrack)
Adds channel channelname to the list of subscribed channels. If streaming has begun, this method will trigger a disconnect/reconnect to pick up the new channel. Will throw an error if the channel is already subscribed. backtrack is the number of previous messages to load when Meteor initially subscribes to the channel. Set to a non-numeric value to load all known events on the channel. Note: backtracking a new channel will not affect any ongoing subscriptions to other channels: the backtrack value is channel specific.
leaveChannel(channelname)
Drops channel channelname from the current Meteor channel list, and triggers a disconnect/reconnect to update the stream. Will throw an error if channelname is not in the channel list.
connect()
Establishes or re-establishes a connection with the Meteor server and begins streaming or polling, as appropriate.
disconnect()
Terminates streaming. Channel list is maintained and streaming will resume at the point it left off when connect() is next called.
registerEventCallback(evt, funcRef)
Maps a local function to a Meteor event. See below for details.

Events

process
Fired when an event message arrives. Arguments returned: data (String)

reset
Fired when Meteor reconnects a dropped stream. Arguments returned: none

eof
Fired when Meteor receives an end-of-file notification from the server (normally happens when the user opens a new window which initiates its own stream. Meteor will not allow any one user to have more than one active stream, so will send an eof() to the old one). Arguments returned: none

changemode
Fired when the interaction mode changes (currently only from stream to poll). Arguments returned: newmode (String)

Example

// Set this to something unique to this client
Meteor.hostid = '409897502705';

// Uncomment to use SSL if you are using https; default http
// Meteor.scheme= "https";

// Our Meteor server is on the data. subdomain
Meteor.host = "data."+location.hostname;

// Call the test() function when data arrives
Meteor.registerEventCallback("process", test);

// Join the demo channel and get last five events, then stream
Meteor.joinChannel("demo", 5);
Meteor.mode = 'stream';

// Start streaming!
Meteor.connect();

// Handle incoming events
function test(data) { window.status = data };

Get it now

Current release

The very latest code is available in the Github repository. However, this is not recommended to most users, and the package offered on this page is considered stable and suitable for public release.

For installation instructions please see the installation page.

License

Meteor is free software licenced under the GNU General Public Licence 2.0. This means that you can:

  • run the program for any purpose;
  • study how the program works and adapt it to your needs;
  • redistribute copies;
  • improve the program and release your improvements to the public

However, you must:

  • keep copyright notices intact and include the licence with any copy of the software that you distribute;
  • only distribute the software under the GPL;
  • make any changes that you make to the software publicly available in source code form under the same terms as Meteor itself.

Release notes

2 March 2008: version 1.06
Fixed: consistent formatting of SHOWSTATS and LISTCHANNELS output
Fixed: send eof message when client reconnects (corrects behaviour when user opens multiple windows)
Fixed: Incorrect closure of new connection if previous connection for same client was waiting on write buffer
Fixed: channel naming made more lenient
Fixed: Revert to poll mode if unable to load forever frame
Added: version number output
Added: configurable transport modes, removed all hardcoded transport-related behaviour
Added: channel info summary, allows polling clients to know where to resume from (+support in the JS class)
Added: output of message ID to event controller after adding a new message
Added: Apache-style logging of 'joinchannel', 'leavechannel' and 'document' events, with useful info such as IP and user agent
Added: Experimental UDP controller server
Added: crossdomain.xml file for flash clients (flash client coming soon)
Added: timestamps to log output, with configurable time formatting
Improved: Updated sample config file to reflect configurable transports syntax
2 February 2008: version 1.05
Fixed: Connections may have been closed while enumerating the Connections array, causing errors when attempting to call a method on an undefined value.
Fixed: Statistics for current_subscribers, current_controllers
Added: documents_not_found statistic
Fixed: Removed null byte message terminators that were causing problems in IE. Message formatting will be made configurable in the next release.
Fixed: Allowed uppercase channel names
Fixed: Simplified channel definition parsing
Fixed: Removed unnecessary document.domain switching
Fixed: Iframe duplication
20 December 2007: version 1.04
Improved: Better Opera and Safari support
Improved: Completely rewritten JS client
Fixed: Content-type fix for serving static .js files
13 Sep 2007: version 1.03
Fixed: Mitigated javascript memory leak in IE
Added: Opera support
Added: Server stats
5 May 2007: version 1.02
Added: multiple channel support
Improved: Changed streaming technique in FF/Saf to the double IFRAME trick
Fixed: Stream reset did not work in IE.
21 Nov 2006: version 1.01
Fixed: Corrected some line endings and file headers
1 Nov 2006: version 1.0
Our first release!

Installation

1. Download and unpack

This installation will assume that you're running some flavour of linux with perl 5 already installed. Create a directory from which you want to run Meteor (/usr/local/meteor is suggested), cd into it, and issue the following command to download the latest source, unpack it, and delete the tar file:

wget http://meteorserver.org/download/latest.tgz
tar zxvf latest.tgz
rm latest.tgz

You will then have the following in your /usr/local/meteor directory:

  • Meteor/ - Meteor's perl modules
    • Channel.pm - A Meteor channel
    • Config.pm - Meteor configuration handling
    • Connection.pm - Common super-class for controller and subscriber
    • Controller.pm - A Meteor controller
    • Document.pm - Caching and serving mechansim for static documents
    • Message.pm - Meteor message object
    • Socket.pm - Meteor Socket additions
    • Subscriber.pm - A Meteor subscriber
    • Syslog.pm - Convenience interface to Sys::Syslog
  • public_html/ - document root for static page serving
    • poll.html - JavaScript IFRAME source for polling connections
    • stream.html - JavaScript IFRAME source for streaming connections
    • meteor.js - JavaScript class required for Meteor web browser client
  • meteord - The Meteor executable
  • meteor.conf.dist - Sample configuration file
  • daemoncontroller.dist - Meteor deamon init script shell script

2. Configure

Copy the init script to /etc/init.d/meteord:

cp daemoncontroller.dist /etc/init.d/meteord

You will need to edit the file to change the path if you did not install meteor in /usr/local/meteor. If you wish to use this to start/stop Meteor, you will need to edit line 14 to specify which user account will be used to run it. The default is meteor, so if you want to create that user account now, type:

useradd meteor

Now copy the configuration file to /etc/meteord.conf:

cp meteord.conf.dist /etc/meteord.conf

If you want to, open up the file and have a look (see the server documentation for more details). Don't edit anything at this stage otherwise it'll muck up the rest of this tutorial.

If you want Meteor to start on system boot, run the following command:

chkconfig meteord on

Help! I did that and Meteor is still not running on boot

If you find that Meteor isn't running automatically on boot, and Meteor's log output says 'sudo: sorry, you must have a tty to run sudo', then you need to comment out the following line in /etc/sudoers:

Defaults    requiretty

This appears to be a change in behaviour between Fedora Core 5 and Fedora Core 6, though of course may affect other Linux distributions.

Help! I get -bash: ./meteord: Permission denied

That's because, for some reason, meteord is not executable. Run the following command to fix that:

chmod +x meteord

3. Run Meteor

Start Meteor in the foreground and get debug output:

cd /usr/local/meteor
./meteord -d

For reference, to manually start Meteor as a background process, you'd do this:

/etc/init.d/meteord start

Help! When I do that I get /etc/init.d/functions: No such file or directory

All our development is done on Fedora, which includes a functions file to enable startup scripts to be a bit prettier. Frankly, all it provides is the ability to print [ OK ] in green, or [ Failed ] in red.

Suggest you simply update the script such that line 6 is removed, and any reference to "&& success || failure" is removed, or use whatever system is preferred on your chosen linux distribution (and if you do, let us know and we'll document it here).

4. Test subscriber connections

Before you start setting up your web pages with the Meteor JavaScript client, take a minute to check that the server is running. Open up your favourite terminal program (we use PuTTY) and connect to the IP on which Meteor is listening (any IP or hostname that resolves to the machine on which Meteor is running), using port 4670. Ensure you're making a raw TCP connection - in PuTTY, select 'Raw' as the protocol. Type the following line and press enter twice:

GET /push/1/iframe/test HTTP/1.1

If all is well you should see the following output:

HTTP/1.1 200 OK
Server: meteord
Content-Type: text/html; charset=utf-8
Pragma: no-cache
Cache-Control: no-cache, no-store, must-revalidate
Expires: Thu, 1 Jan 1970 00:00:00 GMT

<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta http-equiv="Cache-Control" content="no-store">
<meta http-equiv="Cache-Control" content="no-cache">
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Expires" content="Thu, 1 Jan 1970 00:00:00 GMT">
<script type="text/javascript">
window.onError = null;
var domainparts = document.domain.split(".");
document.domain = domainparts[domainparts.length-2] + "." + domainparts[domainparts.length-1];
parent.Meteor.register(this);
parent.Meteor.setServerTime(1161813100);
</script>
</head>
<body onload="r()">
<script>p(-1,"");</script>
<script>p(-1,"");</script>
<script>p(-1,"");</script>
...

You should also see an extra ping snippet (<script>p(-1,"");</script>) being appended about every 3 seconds. Leave it running. You'll also see something like this appear in the shell in which you're running Meteor in debug mode:

meteord(debug): New Meteor::Subscriber for X.X.X.X
meteord(debug): New channel test

5. Test controller connections

Now check that you can inject a message successfully onto the test channel. Open another terminal window and connect to Meteor on its controller interface (again, any IP or hostname that resolves to the machine on which Meteor is running), this time on port 4671. Type the following command:

ADDMESSAGE test hello world

Meteor should respond OK and if you've still got your subscriber connection open, you should see the following message suddenly appear in that window:

<script>p(0,"hello world");</script>

Voila. You just made Meteor stream a message to you. Finally, you'll also see stuff in the debug output of the shell in which you're running Meteor:

meteord(debug): New Meteor::Controller for X.X.X.X
meteord(debug): New message 0

For full details of the event controller command protocol, see the server documentation. To create a real-life useful event controller, you just need to write a program to make a socket connection as you just did with your terminal, and fire ADDMESSAGE commands down it. See the examples page for some complete examples and code.

You can close the terminal sessions now, and stop Meteor.

6. Integrate with web server

Here's the tricky bit. You'll want to serve your website on port 80, obviously, and to avoid cross-site scripting restrictions, Meteor must also serve on the same port. So you have three possible approaches to making this happen:

  1. Set SubscriberPort to 80 and SubscriberIP to a specific IP, say X.X.X.1, in the Meteor config file. This will bind Meteor to a particular IP address on port 80 and leave Apache to bind to port 80 on a different IP. To make this work you will need to run Meteor as root, and configure Apache (or whatever web-server you're using) to bind to port 80 on a different IP, say X.X.X.2.
  2. Configure a firewall to redirect traffic on port 80 of one specific IP, to the Meteor Subscriber port (typically 4670). Then both Meteor and Apache can run in their default configurations, and Meteor can be run as a non-privileged user. So, if you're using iptables you need the following commands:
    iptables -t nat -I PREROUTING -p tcp --dport 80 --destination X.X.X.1 -j REDIRECT --to-port 4670
    iptables -A INPUT -p tcp --dport 4670 -j ACCEPT
    /etc/init.d/iptables save
    
    or if you've got a hardware firewall, set up an equivilent set of rules on that.
  3. Run Meteor on a different physical server to the one running Apache. Then each server need only have one IP, and Apache can be run in its default configuration. Meteor must be set to use port 80 and be run as root.

Since it's likely that the events that you are streaming to your users will be generated by other users of the site, there is a strong argument for having the Meteor server on the same physical machine as your web server. That is, unless you're operating a major site with a multiple server architecture, in which case you can choose whether to have Meteor on each web cluster machine, or have a separate cluster of Meteor servers.

Help! How do I assign multiple IP addresses to a single network card?

Windows, Linux, and Mac OS X all support the assignment of multiple IP addresses to a single NIC. We're assuming you're using Linux, so this is actually pretty easy, just add more addresses using ifconfig:

ifconfig eth0:0 X.X.X.1
ifconfig eth0:1 X.X.X.2

Provided that both addresses are within your machine's subnet, you should be sorted.

7. Configure DNS

Set up your domain's DNS records with two records, as in the following example:

example.com A IN Y.Y.Y.Y (IP for Apache)
data.example.com A IN X.X.X.X (IP for Meteor)

The IPs for Meteor and Apache may actually be assigned to the same physical server, which will be the case with either option (1) or (2) above, but it's vital that the requests are separated out by IP so that the correct server receives them.

Of course you will probably also want a www. record, and that can simply resolve to Apache as well. The data. subdomain in this example assumes that you then set the host property of the Meteor JavaScript client to data.example.com. You can actually use any host that you want, provided that you set the value of the JavaScript host property to the same thing, and it's a subdomain of your Apache host.

8. Test static file serving

Whilst most of the web client javascript Meteor uses is contained in the meteor.js class, there are also specific bits of Javascript that are downloaded by the class at runtime from the Meteor server. It's important to ensure that this process is working, which will confirm that Meteor is running, and that you've set up the SubscriberDocumentRoot config parameter to correctly point at the location of the stream.html and poll.html files (this should be the case if you've done everything as specified in this tutorial).

Open your favourite web browser (or simply use curl) and download (replacing example.com with your domain):

http://data.example.com/meteor.js

You should receive a page of javascript code (you may have to view source to see it).

8. Use Meteor in your web application!

Now that you have Meteor running alongside your preferred web server, all you need to do is create your web application. To include Meteor in your web pages, add the following Javascript include to the header:

<script type="text/javascript" src="http://data.example.com/meteor.js"></script>

Then when you want to start streaming, do something like:

// Set this to something unique to this client
Meteor.hostid = '409897502705';

// Our Meteor server is on the data. subdomain
Meteor.host = "data."+location.hostname;

// Call the test() function when data arrives
Meteor.registerEventCallback("process", test);

// Join the demo channel and get last five events, then stream
Meteor.joinChannel("demo", 5);
Meteor.mode = 'stream';

// Start streaming!
Meteor.connect();

// Handle incoming events
function test(data) { window.status = data };

For full details of the javascript class API, see the JS client documentation. See the examples page for some complete examples and code.


Frequently Asked Questions (FAQ)

  1. How do I run Meteor on port 80?
  2. What cross-site scripting restrictions affect Meteor?
  3. Is caching a problem, and how do you solve it?
  4. Can you stream through a proxy server?
  5. I have a shared/free hosting account. Can I run Meteor?
  6. What can you use Meteor for?

How do I run Meteor on port 80?

To get around cross site scripting restrictions (see What cross-site scripting restrictions affect Meteor?) you'll need to serve Meteor from the same port as your web server. Clearly they can't both bind to the same port on the same IP so you either need to run them on separate machines, or have multiple IPs available to your server. You will probably want to use port 80, since that's the default port used by web browsers, but ports under 1024 are reserved for processes running as root.

So the problem we're addressing here is how to run Meteor on port 80 despite the restriction on running processes on low port numbers. There are two practical solutions: run Meteor as root, or configure your firewall to internally redirect traffic on port 80 of your second IP address to port X, where X is greater than 1024 and is the port on which Meteor is listening.

What cross-site scripting restrictions affect Meteor?

There are three important cross-site scripting restrictions that you should be aware of:

  1. Frames requested from different subdomains of the same parent will not talk to one another until document.domain is set to a domain that both frames share.
  2. Gecko-based browsers will not allow document.domain to be lengthened once it has been shortened.
  3. In Internet Explorer, you can only make XMLHTTPRequests to the host from which the page in which the XMLHTTPRequest object is used was requested.

For a detailed examination of cross site scripting and the same origin policy, see the cross site scripting matrix.

Is caching a problem, and how do you solve it?

Yes, in some cases local caching is an issue, particularly with polling requests. Meteor solves this by appending a millisecond timestamp to the end of each request, making each querystring slightly different. The timestamp parameter is ignored by Meteor server. Meteor also adds no-cache headers to all responses and includes no-cache meta tags (for luck).

Can you stream through a proxy server?

Ish. If there is a proxy server between the client and Meteor, strange things may happen. At worst the client may never receive the stream at all. Some proxies will allow streaming but will have a timeout on the maximum duration of a connection, so the stream will occasionally have to reconnect. Others have no problems at all. If the proxy does not allow streaming the client will give up and use polling mode after the pingtimeout has expired.

I have a shared/free hosting account. Can I run Meteor?

Almost certainly not. Running Meteor requires a fair amount of filesystem surgery, as well as reconfiguration of Apache, added to which you need two IP addresses at your disposal. There may be some ISPs that give you this much freedom within a virtual server account, but we don't know of any.

What can you use Meteor for?

We use it for live chat applications and datalogging, but there are loads of potential uses:

  • Combine it with a bit of clever javascript on the front end and you can draw real-time graphs.
  • Update stock price tables in real time
  • Live update of comments on your blog/forum
  • Display real time telemetry from anything from a remote controlled buggy to supermarket checkouts
  • Real time web analytics (such as the starscape at the top of every page on this site)

Examples

Currently under construction


Live demo

Demo - You're looking at it!

To demonstrate the flexibility and creativity of Meteor, we've hooked it into this website. The stars you can see appearing at the top of your browser are requests to our site, including your own requests. Each star represents one visitor, and the brightness of the star, the time since their last request (bright stars are the most recent requests). Red stars represent errors, eg. the user has requested a page that doesn't exist, so our server has sent back a 404.

Very cool. How does it work?

The server running meteorserver.org has two IP addresses, and has Apache listening on one of them, and Meteor on the other (both on port 80). Apache's IP is listed in DNS as the A record for meteorserver.org, while Meteor's IP is the A record for data.meteorserver.org.

There is a daemon running on the server which uses the unix utility tail to monitor the access log, like this:

tail -fs 0.2 /var/log/httpd/access_log

It also has a persistent local connection to Meteor's controller interface, which is listening on port 12345. Any entries that are made by Apache in the log file are picked up by the daemon and reformatted into JSON as follows:

{ip:'000.000.000.000', path:'/path/to/file/requested.html', respcode:200}

Finally the daemon injects the message into Meteor on a channel called demo, by sending the following command over its connection to Meteor's controller interface:

ADDMESSAGE demo {ip:'000.000.000.000', path:'/path/to/file/requested.html', respcode:200}

On the client side, each webpage you load on meteorserver.org includes the Meteor JS client, and a fairly straightforward lump of code for invoking it and subscribing to the demo channel. We're only dealing with one channel and one type of message here so we just need to get Meteor to execute a function whenever a message arrives.

Meteor.hostid = hostid; // defined earlier
Meteor.host = "data."+location.hostname;
Meteor.registerEventCallback("process", newHit);
Meteor.joinChannel("demo", 5);
Meteor.mode = 'stream';
Meteor.connect();

Since the message that we're sending is in JSON (though it's important to stress that it doesn't have to be) it can be reconstructed into a data structure by evaling it. We then have a javascript data object that contains all the parameters of the message, and then we can present the data however we want.

My God, it's full of stars

Which brings us to the starscape you can see above. Though completely unrelated to Meteor, you may also be interested to know how the starscape is made. The challenges were to get the stars to plot randomly within a non-rectangular area (also avoiding the shooting star), to convince certain browsers to display a content element small enough to look like a star, and to make the stars fade in and out appropriately.

The first challenge is solved by dividing the top area of the page into a grid of 15x15px squares. Assuming 60 squares per row, we can number the squares such that a single index number is all that is needed to determine an x and y position. By creating a list of valid squares, leaving out those that correspond with positions where we don't want to plot any stars, an irregularly shaped plot area can be formed. Then it's simply a case of choosing a random square from the list of valid squares, converting the index into an x and y offset, and then adding a random number of pixels between 0 and 15 to each dimension to position the star at a random location within the chosen square.

On the second point, Firefox has no problems at all displaying a 1x1px div, but IE tends to moan about it, so each star is actually three nested block elements. The outermost one is also slightly bigger to give us a decent sized target for mouseovers.

Fading is a bit of a pain given that it requires iterative code and timing, so rather than reinventing the wheel we're using the excellent jquery library, which makes it ridiculously easy to fade pretty much any DOM element.

Every so often we run an update process over the array of stars and cull off any that haven't made any requests for a while. The expired stars fade to black and are then removed from the DOM. Enjoy!

"Test" stars

We realise that most of the time there won't be hundreds of people actively navigating the site, so just to make sure that there are some stars (other than the one that represents you), our access log monitor also rendomly injects phantom requests from IPs in the range 10.0.0.1 - 10.0.0.10. Just so you know we're not cheating!


How to join in

Want to help?

What a brave and generous soul you are! Come and suggest to us on the issues. Anyone is welcome to suggest ideas and code to Meteor, and we are actively looking for more people to enrich the project.

If you keen to contribute, please Fork me on Github