The Methabot Project

about

Tue, 23 Dec 2008 15:10:34 +0100

Methabot is targeted for extensibility and customization. It's being developed for high modularity, and comes with javascript as scripting language. With the use of the module system and scripting language, users are able to take full or partial control of the crawling process and decide however Methabot should store web data, statistics and much more.

building

Wed, 31 Dec 2008 20:06:51 +0100

Once you have downloaded Methabot it is time to build and install it. If you got your copy from our subversion repository, be sure to run autogen.sh before continuing. You will need libcurl and SpiderMonkey installed before continuing. First of all you must configure the package for your system, do this by invoking the configure script:

developers

Tue, 23 Dec 2008 14:44:07 +0100

Core Team * Emil Romanus Contributors * Rasmus Karlsson

download

Mon, 23 Feb 2009 22:47:47 +0100

Have a look at this page if you need help with building Methabot. Latest Methabot release is: Methabot/1.6.0.1 Source Code Packages * Methabot/1.6.0.1, Feb 23 2009, Release notes * Methabot/1.6.0, Feb 21 2009, Release notes * Methabot/1.5.0, Jan 15 2009, Release notes * Methabot/1.4.1, Jan 2 2009, Release notes * Methabot/1.4.0, Dec 24 2008, Release notes

e4x

Sat, 29 Nov 2008 21:09:53 +0100

E4X is an extension to javascript. E4X is short for ECMAScript for XML, and allows the scripter to easily access and manipulate XML data. The Methabot project uses E4X as part of it's scripting language for parsers. HTML code is converted into well-defined XML and sent to user-scripted filetype parsers.

faq

Thu, 26 Feb 2009 21:35:57 +0100

* General * * * * The Command Line Tool * * * Configuration Files * * * * E4X Scripting * * General What is The Methabot Project and Methanol? To make a long story short, the Methabot project is an open source project spanning four child projects; The Methabot command line utility, a web crawling library (libmetha), a web crawling daemon, and a search engine server. The name Methanol applies to whenever the client daemon is used in combinatio…

getting_started

Tue, 23 Dec 2008 14:40:21 +0100

So, you've downloaded and installed Methabot? If not, then move back to downloading and building. Once you've installed Methabot, please pick a topic below. Get Familiar with Methabot! * Running Methabot and Tuning it from Command Line * Configuration file basics * How To-index

index

Tue, 23 Jun 2009 16:32:49 +0100

Methabot is an open source web crawler and command line tool optimized for speed. It supports scripted filetype parsing, a wide variety of customization options and is easily configured to fit anyones particular needs. WEBSITE MOVED: This project has moved to a new website:

license

Sun, 27 Jul 2008 22:43:33 +0100

Copyright (c) 2008, Emil Romanus Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies. THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,…

option_reference

Mon, 16 Feb 2009 13:23:53 +0100

Short Option Long Option Parameters Description -M --mode aggressive,friendly,coward Set the amount of time Methabot should wait between all network communication. Default is aggressive. -D --depth-limit (int) Decides how deep Methabot will crawl -e --external If set, external URLs will not be discarded, temporarily disabled -j --jail Restrict the crawling to only subfolders --spread Spread w…

running_methabot

Fri, 16 Jan 2009 10:29:23 +0100

Running methabot is quite simple, when you've successfully installed methabot all you have got to do is decide what you want to do! Default configuration files The first thing you should know is how to load any of the default configuration files. This is easily done by prefixing the name of the configuration file with a colon. For a complete list of default configuration files, you can run methabot with the '--info' flag:

site

Thu, 08 May 2008 18:05:58 +0100

This site/wiki is powered by DokuWiki Release 2008-05-05. It uses a heavily modified version of the Nucleus template, and the original template can be found here.

support

Wed, 07 Jan 2009 04:17:08 +0100

Did you check out the documentation and FAQ before going to this page? Help If you need help, there are multiple options. I recommend you to subscribe to one of the mailing lists. If that doesn't fit you, then try the forum at sourceforge. If you want to report bugs or request features, use the bug tracker.

umex

Sun, 07 Dec 2008 02:37:15 +0100

UMEX is short for URL Matching Expressions. UMEXs provides easy ways of filtering URLs, while avoiding the complexity of regular expressions. Here is an example of a UMEX matching all filenames ending with '.html'. FILE<*.html> And here is a more advanced example matching all URLs with hostname “example.com”, and a file named “test.html” in any subdirectory of /example/.