Google Analytic E-commerce tracking

Useful tool, Modify it to track click (order), By E Commerce module.

Noted: It could set product pricing, But I lazy to do -_-'

Good to know Where your customer came from
Which country order / Sale < can do if modify some script to send product pricing, I set all to 1 usd then Could track only order

From last articles about proxy SEO, I have some research more about this topic and found more keywords interesting, scrape,glype,web crawler perhap it could use with some of our site. Modify to white hat to make webmaster work easier for manage site.

What is it ?

Web scraping (also called Web harvesting or Web data extraction) is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the Web by either implementing low-levelHypertext Transfer Protocol (HTTP), or embedding certain full-fledged Web browsers, such as the Internet Explorer (IE) and the Mozilla Web browser. Web scraping is closely related to Web indexing, which indexes Web content using a bot and is a universal technique adopted by most search engines. In contrast, Web scraping focuses more on the transformation of unstructured Web content, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to Web automation, which simulates human Web browsing using computer software. Exemplary uses of Web scraping include online price comparison, weather data monitoring, website change detection, Web research, Web content mashup and Web data integration.

Web scraping is the process of automatically collecting Web information. Web scraping is a field with active developments sharing a common goal with the semantic Web vision, an ambitious initiative that still requires breakthroughs in text processing, semantic understanding, artificial intelligence and human-computer interactions. Web scraping, instead, favors practical solutions based on existing technologies even though some solutions are entirely ad hoc. Therefore, there are different levels of automations that existing Web-scraping technologies can provide:

Techniques for Web scraping

• Human copy-and-paste: Sometimes even the best Web-scraping technology can not replace human’s manual examination and copy-and-paste, and sometimes this may be the only workable solution when the websites for scraping explicitly setup barriers to prevent machine automation.
• Text grepping and regular expression matching: A simple yet powerful approach to extract information from Web pages can be based on the UNIX grep command or regular expression matching facilities of programming languages (for instance Perl orPython).
• HTTP programming: Static and dynamic Web pages can be retrieved by posting HTTP requests to the remote Web server using socket programming.
• DOM parsing: By embedding a full-fledged Web browser, such as the Internet Explorer or the Mozilla Web browser control, programs can retrieve the dynamic contents generated by client side scripts. These Web browser controls also parse Web pages into a DOM tree, based on which programs can retrieve parts of the Web pages.
• HTML parsers: Some semi-structured data query languages, such as the XML query language (XQL) and the hyper-text query language (HTQL), can be used to parse HTML pages and to retrieve and transform Web content.
• Web-scraping software: There are many Web-scraping software available that can be used to customize Web-scraping solutions. These software may provide a Web recording interface that removes the necessity to manually write Web-scraping codes, or some scripting functions that can be used to extract and transform Web content, and database interfaces that can store the scraped data in local databases.
• Semantic annotation recognizing: The Web pages may embrace metadata or semantic markups/annotations which can be made use of to locate specific data snippets. If the annotations are embedded in the pages, as Microformat does, this technique can be viewed as a special case of DOM parsing. In another case, the annotations, organized into a semantic layer, are stored and managed separated to the Web pages, so the Web scrapers can retrieve data schema and instructions from this layer before scraping the pages.

Legal issues

Web scraping may be against the terms of use of some websites. The enforceability of these terms is unclear. While outright duplication of original expression will in many cases be illegal, in the United States the courts ruled in Feist Publications v. Rural Telephone Service that duplication of facts is allowable. Also, in a February, 2006 ruling, the Danish Maritime and Commercial Court (Copenhagen) found systematic crawling, indexing and deep linking by portal site ofir.dk of real estate site Home.dk not to conflict with Danish law or the database directive of the European Union.

U.S. courts have acknowledged that users of "scrapers" or "robots" may be held liable for committing trespass to chattels, which involves a computer system itself being considered personal property upon which the user of a scraper is trespassing. However, to succeed on a claim of trespass to chattels, the plaintiff must demonstrate that the defendant intentionally and without authorization interfered with the plaintiff's possessory interest in the computer system and that the defendant's unauthorized use caused damage to the plaintiff. Not all cases of web spidering brought before the courts have been considered trespass to chattels.

In Australia, the Spam Act 2003 outlaws some forms of web harvesting.

Technical measures to stop bots

The administrator of a website can use various measures to stop or slow a bot. Some techniques include:

• If the application is well behaved, adding entries to robots.txt will be adhered to. Google and other well-behaved bots can be stopped this way.
• Blocking an IP address. This will also block all browsing from that address.
• Sometimes bots declare who they are. Well behaved ones do (for example 'googlebot'). They can be blocked on that basis. Unfortunately, malicious bots may declare they are a normal browser.
• Bots can be blocked by excess traffic monitoring.
• Bots can be blocked with tools to verify that it is a real person accessing the site, such as the CAPTCHA project.
• Sometimes bots can be blocked with carefully crafted Javascript.
• Locating bots with a honeypot or other method to identify the IP addresses of automated crawlers.

• Screen scraping
• Scraper site
• Mashup (web application hybrid)
• Text corpus
• Corpus linguistics

Some interesting article about how to scrape site
from oooff

Most Basic Web Data Parsing Script

Whole script -

The whole script minus the line numbers of course. Those are just their for our reference.

1. <?php 
2. $data = file_get_contents('http://search.msn.com/results.aspx?q=site%3Afroogle.com');
3. $regex = '/Page 1 of (.+?) results/';
4. preg_match($regex,$data,$match);
5. var_dump($match); 
6. echo $match[1];
7. ?>

Script Explanation -

Ok here goes with the basic explanation…

Line 2.
$data = file_get_contents('http://search.msn.com/results.aspx?q=site%3Afroogle.com');
Now if you studied up on the first tutorial you'll know that we're pulling data from MSN search using the file_get_contents command and assigning the data to the $data variable.

However we're also passing some data in the url to get the specific page from MSN that we want to scrape. If you already know about passing variables in the url you can go to Line 3.

You might be asking what is all that stuff after the MSN url? I'm sure you've seen it a lot of times but might not been sure what it was. Basically what all that stuff is, is just like passing a variable in a php script but you're doing it through a url. Lets take a peak at the url we're using here to get a better understanding. Our url if you don't remember is "http://search.msn.com/results.aspx?q=site%3Afroogle.com".

Let's break it into two parts split on the question mark. Why you ask? That's where the url ends and the data being passed begins. With is separated we have:

http://search.msn.com/results.aspx
and
q=site%3Afroogle.com

Now I hope I don't need to go into an explanation on the first part so I'm really only going to talk about the second. Also I'll do some basic tutorials on accepting data later so you have an understand what happens to this url on the other side. When you look at the second part of the url you'll always see a field and a value for the field, although sometimes that value is blank. How do you know which is the field and which is the value you ask? The field is always going to come before the equal sign = and the value will come after. Basically think of it like assigning a variable a value. In this data being passed by the url our field is "q" if you didn't already guess and our value is site%3Afroogle.com. The field 'q' that MSN takes stands for query. So passing data assigned to the 'q' field is telling MSN, "hey look this search/query up for me."

The value assigned to the field 'q' is site%3Afroogle.com. First thing you're probably thinking is what in the world is that %3A, I didn't type that. Well to keep things very simplistic, there's certain variables that can't be passed through url's things like colon's, quotes, semi-colon's etc, because these are protected and mean certain things to a web server when they see them. So we need to use some other form of formatting. In this case we're converting the ':' in site:froogle.com to a encoded value (more on that later). So what we're asking for by the site: command in MSN is how many pages from site X are in your search engine. So specifically how many pages from froogle are indexed in MSN.

Click here to see the page we're scraping

// Some hosting aware file_get_contents as danger commmand, Maybe we should try curl, or some class.

Line 3.
$regex = '/Page 1 of (.+?) results/';

div id="search_header"><h1>site:froogle.com</h1><h5>Page 1 of 9,138 results</h5>&nbsp;<b>&#01

Anytime you see a $varname = 'something here'; or $varname = "something here"; you know it's just a value being assigned to a variable. Also note you can use single ' and double " quotes interchangeably.

(.+?) is our best friend when it comes to regex, it basically means match everything starting from the text ( I'll call that text anchors too, so be prepared for me to use the interchangeably) in the beginning and stopping at our end text/anchor. Something like this:

opening anchor text here ( .+?) closing anchor text here

Pretty easy huh? Yeah I thought so. The only other thing to note in this is that there is the forward slashes in the '/stuff/'; that's a regex thing. Just know that in php you always need to let regex know what to match inside of forward slashes.

// Know more about regex here : http://de.php.net/manual/en/reference.pcre.pattern.syntax.php

Line 4.

preg_match($regex,$data,$match);
Ah a new function's in town, preg_match(). Preg_match() is the PHP function to call regex for a single match. So anytime we want to match one thing in our data we're going to call the parsing function preg_match().

With preg match we're doing something called passing data to the function for it to work on. In this case we're passing $regex, $data, $match. We know what both $regex (parsing string we just made) and $data (scraped page from MSN) are but what is the $match variable? It's just the variable that our parsed data is going to be returned to. In plain english we're saying take $data and then apply the filter $regex to it. Then whatever comes through that filter dump out into $match. Make sense?

I sure hope you said yep, that's easy.

Line 5.
var_dump($match); 
The function var_dump() is your best friend as a programmer. It says whatever is in this variable or array dump it out onto the screen so I can see what's happening. So this line will output this onto the screen.

array(2) {
[0]=>
string(23) "Page 1 of 9,138 results"
[1]=>
string(5) "9,138"
}

Array? What's that? Well this is as good a time as any to introduce what an array is. They're extremely useful tools for you to know. So lets backup a little we know that a variable is something that holds 1 thing, right? Well an array is just like a variable except it holds multiple things. I like to think of it like this. Stop and imagine a train for a second it has all these cars on it that hold things right? well a variable is a single car and can only hold a single thing. Where an array is like a train that has multiple cars to hold things. In the output above we have a two (2) cell array, which is just like a 2 car train. In car 0 we have the string 'Page 1 of 9,138 results' and in car 1 we have the string '9,138', which is the result we want right? You might be asking why does preg_match return an array rather then just a simple string. It does this two give you two options on how to match things. You'll notice car/cell 0 has the anchors included as well as the matched text. Where car 1 only has the text inside the anchors. 

Line 6.
echo $match[1];
What's with the new notation? If you hadn't already guessed that's how we access the cars in our train. We know if we have a array and what we want is in car 1 we access that by 'referencing' that car which is what the [1] means. We want to output only what's in the second cell because we don't want the anchors included. This will output to our screen:

9,138

Which is exactly what we aimed to do.

Other things to try -

So fun stuff to try using our new skills.

1. Use the link: command in MSN and see if you can get the number of links for a domain. Don't forget that : = %3A

2. See if you can get the title of a web of any web page. Hint: anchors are going to be <title> and </title>.

Conclusion -

You can make some pretty cool tools with just the two very basic things I've shard with you so far. Pulling data from somewhere using the file_get_contents() function and the data parsing preg_match() function. Have fun with it and I'll see you on the next data scraping tutorial.

wiki scrape

<?php
function wikipedia($article)	{
	$pattern[0] = '/<a href="(.*?)">(.*?)<\\/a>/';
	$replace[0] = '$2';
	$pattern[1] = '/<h3 id=\"siteSub\">From Wikipedia, the free encyclopedia<\/h3>/';
	$replace[1] = '';
	$pattern[2] = '/<div id=\"contentSub\">(.*?)<\/div><div id=\"jump-to-nav\">Jump to: navigation, search<\/div>/';
	$replace[2] = '';
	$pattern[3] = '/<div class=\"messagebox cleanup metadata\">(.*?)<p><br \/><\/p>/';
	$replace[3] = '';
	$pattern[4] = '/<table class=\"messagebox\" (.*?)>(.*?)<\/table>/';
	$replace[4] = '';
	$pattern[5] = '/<dl>(.*?)<\/dl>/';
	$replace[5] = '';
	$pattern[6] = '/<h1 class=\"firstHeading"\>(.*?)<\/h1>/';
	$replace[6] = '<h3>$1</h3>';
	$pattern[7] = '/<table class=\"messagebox protected\" style=\"border: 1px solid #8888aa; padding: 0px; font-size:9pt;\">(.*?)<\/table>/';
	$replace[7] = '';
	$pattern[8] = '/<div class=\"infobox sisterproject\">(.*?)<\/div><\/div>/';
	$replace[8] = '';
	$pattern[9] = '/<sup (.*?)>(.*?)<\/sup>/';
	$replace[9] = '';
	$pattern[10] = '/<table style=\"background: transparent;\" width=\"0\">(.*?)<\/table>/';
	$replace[10] = '';
	$pattern[11] = '/<table class=\"messagebox current\" style=\"font-size:	normal;\">(.*?)<\/table>/';
	$replace[11] = '';
	$pattern[12] = '/<table class=\"toccolours\" align=\"center\" width=\"55%\" cellpadding=\"0\" cellspacing=\"0\">(.*?)<\/table>/';
	$replace[12] = '';
	$pattern[13] = '/<div class=\"editsection\"(.*?)>(.*?)<\/div>/';
	$replace[13] = '';
	$pattern[14] = '/<div id=\"bodyContent\">/';
	$replace[14] = '<div>';
	$pattern[15] = '/<dd>(.*?)<\/dd>/';
	$replace[15] = '';
	$pattern[16] = '/<div class=\"messagebox cleanup metadata\">(.*?)<\/div>/';
	$replace[16] = '';
	$pattern[17] = '/<div class=\"thumbcaption\">(.*?)<\/div><\/div>/';
	$replace[17] = '';
	$pattern[18] = '/<div class=\"thumb tright\">/';
	$replace[18] = '';
	$pattern[19] = '/\[(.*?)\]/';
	$replace[19] = '';
	$pattern[20] = '/<table class="messagebox protected" (.*?)>(.*?)<\/table>/';
	$replace[20] = '';
	$pattern[21] = '/<div style="position:absolute; z-index:100; right:20px; top:10px; height:10px; width:300px;"><\/div>/';
	$replace[21] = '';
	$pattern[22] = '/<div style="position:absolute; z-index:100; right:10px; top:10px;" class="metadata" id="administrator">(.*?)<\/div><\/div>/';
	$replace[22] = '';
	$pattern[23] = '/<table class="messagebox current"(.*?)>(.*?)<\/table>/';
	$replace[23] = '';
	$pattern[24] = '/<table class="messagebox current" style="width: auto;">(.*?)<\/table>/';
	$replace[24] = '';
	$pattern[25] = '/<div class="dablink">(.*?)<\/div>/';
	$replace[25] = '';
	$pattern[26] = '/<b>/';
	$replace[26] = '<strong>';
	$pattern[27] = '/<\/b>/';
	$replace[27] = '</strong>';
	$pattern[28] = '/<div(.*?)>/';
	$replace[28] = '';
	$pattern[29] = '/<\/div>/';
	$replace[29] = '';
	$pattern[30] = '/<map(.*?)>(.*?)<\/map>/';
	$replace[30] = '';
	$pattern[31] = '/<img src="(.*?)" alt="This page is semi-protected." width="18" (.*?)\/>/';
	$replace[31] = '';
	$pattern[32] = '/<table style="width:100%;background:none">(.*?)<\/table>/';
	$replace[32] = '';
	$pattern[33] = '/<div class="messagebox merge metadata">(.*?)<\/div>/';
	$replace[33] = '';
	$wikipedia = fopen($article, "r");
	$wikipedia = preg_replace($pattern, $replace, $wikipedia);
		if (preg_match("/<\!-- start content --\>(.*)<table id=\"toc\" class=\"toc\" summary=\"(.*)\">/", $wikipedia, $w)) {
			$wikipedia = $w[1];
		} elseif (preg_match("/<\!-- start content --\>(.*)<a name=\"(.*)\">/is", $wikipedia, $w)) {
			$wikipedia = $w[1];
		} elseif (preg_match("/<\!-- start content --\>(.*)<div class=\"boilerplate metadata\" id=\"stub\">/is", $wikipedia, $w)) {
			$wikipedia = $w[1];
		} elseif (preg_match("/<\!-- start content --\>(.*)<div class=\"printfooter\">/is", $wikipedia, $w)) {
			$wikipedia = $w[1];
		}
	}
	print $wikipedia;
}
?>

very good article from seofaststart

How To Tell If You've Been Proxy Hacked

The simplest test, if you are experiencing a problem, is to examine Google search results for a phrase (search term in quotes) that should be unique to your page. For example, if your home page says "Fred's Widget Factory sells the best down-home widgets on Earth" then you can search for that phrase.

You want to use a phrase (or combination of phrases) that should only appear on your page, and nowhere else on the web… or very few places at least. Then you do the search – if there's more than one result (your page), then you need to examine the other URLs that are listed. If some of them are delivering an exact copy of the page, you just may be dealing with a proxy that has hijacked your content.

A typical proxy link looks something like this:
www.example.com/nph-proxy.pl/011110A/http/www.mattcutts.com/blog/
It's easy to see what URL that would fetch, if example.com were a real proxy. Other proxy URLs encode the target URL so it's not always that easy to determine what they're going to fetch just by looking.

The mere presence of proxies in the index doesn't necessarily mean you'll be dropped or penalized. The situation inside Google's systems is no doubt very complex. I have seen sites with multiple proxies indexed, and no ill effects. It's possible that there are certain factors (trust, authority, domain age, etc.) that make one site more susceptible than another. I have no idea how they make the decision on which copy of a page to keep.

Why Is This Even Possible?

In simple terms, it appears that the original (authentic) page gets dropped or penalized as duplicate content.

A couple years ago, Google deployed some software & infrastructure changes collectively known as "Big Daddy." This involved crawling from many different data centers, and changes to the crawler itself. It appears that the changes include moving some of the duplicate content detection down to the crawlers. The bug probably arises from the way the data centers are synchronized. Pure speculation here, but the picture I have of what happens looks like this:

1. The original page exists in at least some of the data centers.
2. A copy (proxy) gets indexed in one data center, and that gets sync'd across to the others.
3. A spider visits the original, checks to see if the content is duplicate, and erroneously decides that it is.
4. The original is dropped or penalized.

As far as whether "any site" could get hacked," I don't know. I'm not a black hat. I don't have a link farm. I don't have a botnet to spam blogs with. So I can't manufacture thousands of links to thousands of proxies, in an attempt to knock sites off of SERPS. I wouldn't do that anyway – it's evil. So what I know is based mostly on sites reporting a problem, blocking the proxies, and seeing the problem disappear after the proxies are gone. Then repeating the exercise with the same results.

How To Fight Back

There are basically three main possibilities for your situation:

Situation 1: You are running an Apache server. We have 2 solutions in this case, that were developed by Jaimie Sirovich (co-author of Professional Search Engine Optimization with PHP). We've worked some late nights on this.

Solution #1 uses mod_write and .htaccess, to pass all spider requests through a PHP script that validates the request. This will only defends against being hacked via "normal" anonymous proxies that pass long the user agent – it only inspects visits from the "Big 4" search engines (Ask, Google, MSN, and Yahoo). I call this the "first tier" defense – it won't stop every proxy that exists, but it will come close, and you can implement it without modifying any of your applications. It wil even work if your web site is all static pages. This is what I'm implementing. Jaimie doesn't like it because it's kind of a hack – and he would rather you didn't use it at all.

Solution #2 is a PHP script that implements the "reverse cloaking" defense, putting a "nonindex, nofollow" robots meta tag into your pages unless it's a spider that you have configured the script to recognize. This will only be possible if your site is built on PHP. It wouldn't be terribly difficult for a competent PHP user to implement this in an all-static site, you'd just need to change .htaccess so that your .html files are parsed as PHP. A WordPress plug-in will follow soon. This is a more robust defense, against more proxies.

How to get the code: An implementation guide is provided on Jaimie's blog, along with a testing environment that you can use to check spider user agents & IP addresses, and of course the source code for both solutions. No warranty is given. This is hard core code for a hard core situation. Don't use it if you don't need it, and all code should really be deployed by professionals who can understand what it does, modify it to suit unique environments, etc.

Situation 2: You are running a Microsoft (IIS) server. Jaimie is working on an IIS/ASP solution similar to the Apache/PHP solution, which should be available soon. Think days, not weeks, in other words. Much sooner than his new book (Professional SEO with ASP), which is also in the pipeline.

Situation 3: You are on a hosted solution, aren't running PHP scripts that you can edit, don't control the web server, etc. This is a more complex situation. I will have another post tomorrow that will offer some possible solutions, including one that involves creating your own caching proxy on a separate server. In this case, I don't recommend doing anything unless you really believe that you have a problem with proxies.

There are other solutions available. Bill Atchison's Crawlwall is a professional (commercial) solution, that does a lot more to prevent content theft, etc. If you have the means, you may want to consider this instead, and move the burden of "keeping up with the spiders" onto Bill's shoulders. Jaimie is working on a more general proxy-blocking solution as well. Ekstreme has the beginnings of a spider validation solution in the PHP Search Engine Bot Authentication code they published.

If You Are Operating A Proxy – Don't Be Part of the Problem

If you are operating a proxy server, and you don't want to be part of the problem, you can prevent your server from being used as a tool by adding a robots.txt file that prevents all search engine spiders from indexing proxied content through your server. For example, if all proxy URLs begin with /proxy/ then you can use:

User-agent: *

Disallow: /proxy/

Of course, not all proxies are being run by innocent people for innocent reasons. Some of them are actually designed to hijack content – to deliver ads, etc. Some people want to steal your content, and they want the search engines to index it. In fact, I would not be surprised if a large part of the overall problem isn't caused by such people firing links at their own proxies.

Is It Just Google?

You got me… I haven't seen any cases on other engines that looked like a proxy hack, but I'd be surprised if it only affected Google. Google may simply be the only search engine that shows you enough search results to let you "catch" the proxies. Google may be more susceptible because they crawl more URLs more often, and use multiple data centers.

Assuming I am not completely wrong, it sure looks like less of a design flaw, and more of an "emergent property" of the very things that make Google the world's best search engine (just my opinion, apparently the average consumer no longer agrees). I don't know that there is an easy solution, especially if the problem arises because of their multiple-data-center strategy.

Unfortunately, any countermeasures that we implement could be thwarted by someone willing to copy our content in other ways, or by constructing a proxy that spoofs user agents, uses intermediate proxies to hide its IP address, and strips out meta tags. This has always been possible, BTW. Anyone actually doing these things, of course, would likely be committing a crime… and would be a lot easier to find than some script kiddie using comment spam to fire links at someone else's proxies.

UPDATE: As of May 1, 2008, I have every reason to believe that Google has solved this problem, at least in the general case. At this point, the only sites I can see getting "duped by proxy" are spammier than the proxies themselves.

Update again: September 2009 - damned if this thing hasn't cropped up again – now it looks like Google's replacing the duped URL with the copy's URL – and even RANKING the duplicates… (similar to the already-known-and-passed-off-as-a-feature 302 redirect bug).

My view from this point, Could use this SEO tip for improve our system, with concern : Google bot not stupid, Perhap it need some tips to make it's work.
For protect this following article is great : SEO image

For those of you who do not know, SEO Image is one of the most plagiarized websites. Our content is stolen and rewritten every day by new and novice SEO companies throughout the world.

One issue we have is that these novice SEO companies not only copy word for word, but some cause the same effect that the proxy search portals do. That is a duplicate content filter. For those of you new to my blog, I have been very anti-duplicate content filter since its unleashing in 2005 as an overly aggressive filter.

So, to take this further the proxy sites are ways that searchers can try and mask the IP they are searching from, as the proxy server will allow someone to access a site that has banned regions from accessing it. I do not want to get to technical with the proxy servers, you can read more at Wikipedia. The problem with proxy servers is that they cache websites that are searched and then allow search engines to spider them so that they can appear as larger websites (page spam) and rank better so that people can click on the paid ads.

The Google URL Removal tool is a sure way of removing proxy duplicates. Since we feel the duplicate content filter will remove most copies, the proxy search results concern us because they are used by Black Hat SEO’s to try and hurt other websites rankings.

There is one easy way to remove the proxy servers with Google’s Remove URL tool. That is, first you need to be able to deny IP ranges from accessing your website in either Windows IIS Administration, or htaccess for Linux Servers.
First Step:

1. Find the proxy indexed in Google with your content
2. Find the Reverse DNS using DNS Stuff to determine the IP we generally block the Name services and the IP by C Class (XXX.XXX.C-Class.XXX). If the IP does not work, try our Server Header Checker Tool.
3. Using your .htaccess file or IIS Administration deny access to the IP ranges by the C Class of the IP.
4. Click the link in the Google Search Results and see if it returns a 403 Forbidden Code.

This is where it gets tricky, if you get the 403 code, then the site will no longer be duplicating you, however, if the site uses a frameset or iframe, then you will NOT be able to use the Google URL Removal Tool as it will see a 200 “Found” header directive and assume the page still exists.

Use the URL Removal Tool and check off “anything associated with this domain”. If the site does not use frames then you will get it removed, if it does have frames then google gets a 200 code and will NOT remove the site despite the frame. You can try to access the frame and submit that page, but it generally will not help.

All in all, the ability of proxy servers to hurt rankings is unknown. We believe it will effect some of the sites rankings, but may not be the full story. Another issue of proxy servers, is that they can 302 hijack sites if they are set poorly.

We have not found any code that can ban proxy servers even ones that use nph-proxy.cgi.

How To Fight Back — Code implementations

Seo-Egg

Well that's where I come in. I have 2 implementations in beta (read: they work according to my tests, but I'm going to be testing more) that address the problem based on the methods the search engines cite. Then, essentially, we're using a benign form of cloaking (yes, cloaking!) to make it more difficult for bad bots, proxies, etc. to exploit us.

I'll expand the explanation in that documentation to make it easier to comprehend/install. But if you know PHP, dive right in.

The code and concepts were primarily based off on the book I coauthored, "Search Engine Optimization with PHP." It is my sentiment that most SEOs have to be aware of technology more so than they think — hence the book authored by me and co-author Cristian Darie. This is just one example.

 

Note: I didn't realize WP changes quotes to curly quotes to look "pretty" since version 2.1. I turned that feature off. Cut and pasting should work.

Below is the main class necessary for the cloaking functionality, "SimpleCloakV2:"

<?php

$__metaRobotsExcludeProxiesCallbackHTML = ";

/*
// +----------------------------------------------------------------------+
// | SimpleCloakV2 Version 2                                                |
// | Class for cloaking content                                           |
// | http://www.SEOEgghead.com                                            |
// +----------------------------------------------------------------------+
// | Copyright (c) 2005-2006 Jaimie Sirovich and Cristian Darie           |
// +----------------------------------------------------------------------+
*/

// load configuration file
require_once('config.inc.php');

class SimpleCloakV2
{  
    
  function _connect()
  {
      if (USE_CUSTOM_CONNECT_CODE) return true;
    // Connect to MySQL server
    $dbLink = mysql_connect(DB_HOST, DB_USER, DB_PASSWORD)
                    or die("Could not connect: " . mysql_error());

    // Connect to the seophp database 
    mysql_select_db(DB_DATABASE) or die("Could not select database");      
    
    return $dbLink;
  }
    
  function _close($dbLink)
  {
      if (USE_CUSTOM_CONNECT_CODE) return true;
    // close database connection 
    mysql_close($dbLink);
  }
  
  // returns the confidence level
  function isSpider($spider_name = ", $check_uas = true, $check_ips = true, $use_user_defined_data = true, $ignore_bad_uas = true)
  {
    // default confidence level to 0
    $confidence = 0;
    
    // matching user agent?    
    if ($check_uas)
      if (SimpleCloakV2::_get(0, $spider_name, 'UA', $_SERVER['HTTP_USER_AGENT'], ", $use_user_defined_data ? " : 'N', $ignore_bad_uas ? 'bad' : ")) 
        $confidence += 2;
  
    // matching IP?    
    if ($check_ips)
      if (SimpleCloakV2::_get(0, $spider_name, 'IP', ", $_SERVER['REMOTE_ADDR'], $use_user_defined_data ? " : 'N', $ignore_bad_uas ? 'bad' : ")) 
        $confidence += 3;     
    
    // return confidence level
    return $confidence;
  }
  
  // retrieve cloaking data filtered by the supplied parameters
  function _get($id = 0, $spider_name = ", $record_type = ", 
               $value = ", $wildcard_value = ", $is_user_defined_data = ", $not_spider_name = ")
  {
    // by default, retrieve all records
    $q = " SELECT cloak_data.* FROM cloak_data WHERE TRUE ";
           
    // add filters
    if ($id) {
      $id = (int) $id;
      $q .= " AND id = $id ";
    }     
    if ($spider_name) {
      $spider_name = mysql_escape_string($spider_name);
      $q .= " AND spider_name = '$spider_name' ";
    }      
    if ($record_type) {            
      $record_type = mysql_escape_string($record_type);
      $q .= " AND record_type = '$record_type' ";
    }      
    if ($value) {  
      $value = mysql_escape_string($value);
      $q .= " AND value = '$value' ";
    }
    if ($wildcard_value) {
      $wildcard_value = mysql_escape_string($wildcard_value);
      $q .= " AND ( '$wildcard_value' = value OR '$wildcard_value' LIKE CONCAT(value, '.%') ) ";
    }
    if ($is_user_defined_data) {
      $is_user_defined_data = mysql_escape_string($is_user_defined_data);
      $q .= " AND is_user_defined_data = '$is_user_defined_data' ";
    }
    if ($not_spider_name) {
      $not_spider_name = mysql_escape_string($not_spider_name);
      $q .= " AND spider_name <> '$not_spider_name' ";
    }      
    
    $dbLink = SimpleCloakV2::_connect();
          
    // execute the query
    $tmp = mysql_query($q);
    
    SimpleCloakV2::_close($dbLink);
      
    // return the results as an associative array
    $rows = array();
    while ($_x = mysql_fetch_assoc($tmp)) {
      $rows[] = $_x;
    }
    return $rows;
  }  
  
  // updates the entire database with fresh spider data, but only if our data is
  // more than 7 days old, and if the online version from iplists.org has changed
  function updateAll($delete_user_defined_data = false)
  {      

      $dbLink = SimpleCloakV2::_connect();
      
    // retrieve last update information from database
    $q = "SELECT cloak_update.* FROM cloak_update";
    $tmp = mysql_query($q);
    $updated = mysql_fetch_assoc($tmp);    
    $db_version = $updated['version']; 
    $updated_on = $updated ['updated_on'];
 
    // get the latest update more recent than 7 days, don't attempt an update    
    if (isset($updated_on) && 
        (strtotime($updated_on) > strtotime("-604800 seconds"))) 
    {
      // close database connection 
      SimpleCloakV2::_close($dbLink);
      // return false to indicate an update wasn't performed
      return false;
    }
 
    
    // read the latest iplists version     
    $version_url = 'http://www.iplists.com/nw/version.php';
    
    $ch = curl_init();    
    curl_setopt ($ch, CURLOPT_URL, $version_url);
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt ($ch, CURLOPT_TIMEOUT, 60);
    $latest_version = curl_exec($ch);
    curl_close($ch);
    
    $latest_version = mysql_escape_string($latest_version);

    // if no updated version information was retrieved, abort
    if (!$latest_version) 
    {
      // return false to indicate an update wasn't performed
      return false;
    }
    
    // save the update data
    $q = "DELETE FROM cloak_update";
    mysql_query($q);
    $q = "INSERT INTO cloak_update (version, updated_on) " . 
         "VALUES('$latest_version', NOW())";
    mysql_query($q);
    
    // if we already have the current data, don't attempt an update 
    if ($latest_version == $db_version) 
    {
      // close database connection 
      mysql_close($dbLink);
      // return false to indicate an update wasn't performed
      return false;
    }
 
    // update the database
    SimpleCloakV2::_updateCloakingDB('google', 
                                  'http://www.iplists.com/nw/google.txt', $delete_user_defined_data);
    SimpleCloakV2::_updateCloakingDB('yahoo', 
                                  'http://www.iplists.com/nw/inktomi.txt', $delete_user_defined_data);
    SimpleCloakV2::_updateCloakingDB('msn', 
                                  'http://www.iplists.com/nw/msn.txt', $delete_user_defined_data);
    SimpleCloakV2::_updateCloakingDB('ask', 
                                  'http://www.iplists.com/nw/askjeeves.txt', $delete_user_defined_data);
    SimpleCloakV2::_updateCloakingDB('altavista', 
                                  'http://www.iplists.com/nw/altavista.txt', $delete_user_defined_data);
    SimpleCloakV2::_updateCloakingDB('lycos', 
                                  'http://www.iplists.com/nw/lycos.txt', $delete_user_defined_data);
    SimpleCloakV2::_updateCloakingDB('wisenut', 
                                  'http://www.iplists.com/nw/wisenut.txt', $delete_user_defined_data);

    // close connection 
    SimpleCloakV2::_close($dbLink);

    // return true to indicate a successful update
    return true;
  }
  
  // update the database for the mentioned spider, by reading the provided URL
  function _updateCloakingDB($spider_name, $url, $delete_user_defined_data = false)
  {
      
      $ua_regex = '/^# UA "(.*)"$/m';
      $ip_regex = '/^([0-9.]+)$/m';
      
    // use cURL to read the data from $url
    // NOTE: additional settings are required when accessing the web through a proxy
    $ch = curl_init();    
    curl_setopt ($ch, CURLOPT_URL, $url);
    curl_setopt ($ch, CURLOPT_HEADER, 1);
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt ($ch, CURLOPT_TIMEOUT, 60);
    $result = curl_exec($ch);
    curl_close($ch);
       
    // use _parseListURL to parse the list of IPs and user agents
    $lists = SimpleCloakV2::_parseListURL($result, $ua_regex, $ip_regex);
    
    // if the user agents and IPs weren't retrieved, we cancel the update 
    if (!$lists['ua_list'] || !$lists['ip_list']) return;

    // lock the cloack_data table to avoid concurrency problems
    mysql_query('LOCK TABLES cloak_data WRITE');
    
    // delete all the existing data for $spider_name
    SimpleCloakV2::_deleteSpiderData($spider_name, $delete_user_defined_data ? " : 'N');

    // insert the list of user agents for the spider
    foreach ($lists['ua_list'] as $ua) {
      SimpleCloakV2::_insertSpiderData($spider_name, 'UA', $ua);
    }

    // insert the list of IPs for the spider
    foreach ($lists['ip_list'] as $ip) {
      SimpleCloakV2::_insertSpiderData($spider_name, 'IP', $ip);
    }
    
    // release the table lock
    mysql_query('UNLOCK TABLES');
  }
  
  // helper function used to parse lists of user agents and IPs
  function _parseListURL($data, $ua_regex, $ip_regex)
  {
    $ua_list_ret = preg_match_all($ua_regex, $data, $ua_list);
    $ip_list_ret = preg_match_all($ip_regex, $data, $ip_list);
    return array('ua_list' => $ua_list[1], 'ip_list' => $ip_list[1]);  
  }

  // inserts a new row of data to the cloaking table
  function _insertSpiderData($spider_name, $record_type, $value, $is_user_defined = 'N')
  {
    // escape input data
    $spider_name = mysql_escape_string($spider_name);
    $record_type = mysql_escape_string($record_type);
    $value = mysql_escape_string($value);
    $is_user_defined = mysql_escape_string($is_user_defined);

    // build and execute the INSERT query
    $q  = "INSERT INTO cloak_data (spider_name, record_type, value, is_user_defined) " . 
          "VALUES ('$spider_name', '$record_type', '$value', '$is_user_defined')";
    
    mysql_query($q);
  }
  
  // delete the cloaking data for the mentioned spider
  function _deleteSpiderData($spider_name, $is_user_defined = ")
  { 
    // escape input data
    $spider_name = mysql_escape_string($spider_name);

    // build and execute the DELETE query
    $q = "DELETE FROM cloak_data WHERE spider_name='$spider_name'";
    
    if ($is_user_defined) {
      $is_user_defined = mysql_escape_string($is_user_defined);
      $q .= " AND is_user_defined = '$is_user_defined' ";
    }      
    
    mysql_query($q);  
  }
  
  // only use if it's not found via the IPLists cloaking database
  function botVerifyByDNS($ua = array('google', '#.*\.googlebot\.com$#'))
  {
      
      // check cache of bad bots
      if (SimpleCloakV2::isSpider('bad', false, true, true, false)) {
          return false;
      }
      
      // check only UA since this function is only called if the cloaking DB doesn't handle it
      if (SimpleCloakV2::isSpider($ua[0], true, false)) {      
          // reverse lookup
          $host_name = gethostbyaddr($_SERVER['REMOTE_ADDR']);
     
          // if it says it's a certain UA but gethostbyaddr the corresponding domain regex, store it and then abort
          if (!preg_match($ua[1], $host_name)) {
              $dbLink = SimpleCloakV2::_connect();
              SimpleCloakV2::_insertSpiderData('bad', 'IP', $_SERVER['REMOTE_ADDR'], 'Y');
              SimpleCloakV2::_close($dbLink);
              return false;
          }
          
          $connected_ip_address = $_SERVER['REMOTE_ADDR'];
          $host_name_ip_address = gethostbyname($host_name);
          
          // if the connected IP matches the authoritative IP, we have a match
          if ($connected_ip_address == $host_name_ip_address) {
              $dbLink = SimpleCloakV2::_connect();
              SimpleCloakV2::_insertSpiderData($ua[0], 'IP', $_SERVER['REMOTE_ADDR'], 'Y');
              SimpleCloakV2::_close($dbLink);
              return true;
          } else {
              // if it says it's a certain UA, gethostbyaddr says the right thing, but gethostbyname does not
              $dbLink = SimpleCloakV2::_connect();
              SimpleCloakV2::_insertSpiderData('bad', 'IP', $_SERVER['REMOTE_ADDR'], 'Y');
              SimpleCloakV2::_close($dbLink);
              return false;
          }
      }
      
      // it does not even say it's a bot via UA
      return false;
      
  }
  
  function _addMetaRobotsExcludeProxiesCallback($buffer)
  {
      global $__metaRobotsExcludeProxiesCallbackHTML;
      return preg_replace('#</title>#', '</title>' . $__metaRobotsExcludeProxiesCallbackHTML, $buffer);
  }
  
  function metaRobotsExcludeProxies($auto_modify_content = true, $uas = array(array('google', '#.*\.googlebot\.com$#'), array('yahoo', '#.*\.yahoo\.net$#'), array('msn', '#.*\.live\.com$#'), array('ask', '#.*\.ask.com$#') ), $meta_tag = '<meta name="robots" content="noindex,nofollow" />', $passlist_regex = ")
  {
      global $__metaRobotsExcludeProxiesCallbackHTML;
      
      if ($meta_tag)
          $__metaRobotsExcludeProxiesCallbackHTML = $meta_tag;
      
      // if it's on our passlist    
      // ex: #become|lycos|somestupidbot#
      if ($passlist_regex) {
          if (preg_match($passlist_regex, $_SERVER['HTTP_USER_AGENT'])) return false;
      }
          
      foreach ($uas as $u) {
          
          // if it's a bot according to UA, then start to investigate
          if (SimpleCloakV2::isSpider($u[0], true, false)) {
              
              // if it's a bot according to IPLists or our user-defined list
              if (SimpleCloakV2::isSpider($u[0], false, true)) {
                  return false;
              // if it's a bot according to DNS
              } else if (SimpleCloakV2::botVerifyByDNS($u)) {
                  return false;
              // if it's not
              } else {
                  if ($auto_modify_content) ob_start(array('SimpleCloakV2′, '_addMetaRobotsExcludeProxiesCallback'));
                  return true;
              }
                      
          }
          
      }
      
    // it's not a bot according to UA
    if ($auto_modify_content) ob_start(array('SimpleCloakV2′, '_addMetaRobotsExcludeProxiesCallback'));
    return true + 1;        
      
  }
    
}
?>

Save this file as "simple_cloak_v2.php."

You will also need the configuration file (it is referenced in "simple_cloak_v2.php"):

<?php
// defines database connection data
// set to "1″ if you are already connected in your application.
define("USE_CUSTOM_CONNECT_CODE", 0);
// usually localhost
define("DB_HOST", "your_db_host");
// db user 
define("DB_USER", "some_user");
// password
define("DB_PASSWORD", "secret");
//db name
define("DB_DATABASE", "your_db");
?>

Save this as "config.inc.php."

Then, to implement:

Use this SQL to create the database tables needed for the SimpleCloakV2 class

Run the following queries in your mySQL database (using the mysql binary or phpmysqladmin):

CREATE TABLE `cloak_data` (
`id` int(11) NOT NULL auto_increment,
`spider_name` varchar(255) NOT NULL default ",
`record_type` enum('UA','IP') NOT NULL default 'UA',
`value` varchar(255) NOT NULL default ",
`is_user_defined` enum('N','Y') NOT NULL default 'N',
PRIMARY KEY (`id`),
KEY `value` (`value`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

CREATE TABLE `cloak_update` (
`version` varchar(255) NOT NULL default ",
`updated_on` datetime NOT NULL default '0000-00-00 00:00:00′
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

Only if you already have a "cloak_data" table (from our book or a previous version of SimpleCloak on the blog), run this SQL:

ALTER TABLE cloak_data ADD `is_user_defined` ENUM('N','Y') NOT NULL;

Populate the Cloaking database with the data from IPLists.com

Note: This should be run periodically from a cron job to keep the data updated. It will update only once a week regardless. However, you may also put it in the footer of an application.

<?php

// load the SimpleCloakV2 library
require_once 'simple_cloak_v2.inc.php';

// update cloaking data and indicate the success status
if (SimpleCloakV2::updateAll())
{
  echo "Cloaking database updated!";
}
else
{
  echo "Cloaking database was already up to date, or the update failed.";
}

?>

Then pick *1* of the following methods.

Note: Method #2 is a bit of a kludge, as the RewriteMap directive of Apache cannot be used in .htaccess. *It has not been tested extensively yet!*

METHOD NUMBER 1 — PHP Implementation

Place this code at the top of your application (or relevant parts thereof):

<?
    include_once('simple_cloak_v2.inc.php');
    $_x = SimpleCloakV2::metaRobotsExcludeProxies();
?>

The code automatically inserts the meta tag using PHP output buffering. If you want a more custom/efficient solution, that is also possible. See the first parameter of function "metaRobotsExcludeProxies." Set to false, it will not use the output buffering, and you may use the result to effect changes in your application as desired.

METHOD NUMBER 2 — .htaccess Implementation

Place this in your .htaccess file

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} yahoo|slurp|msn|ask|google|gsa [NC]
RewriteRule (^.*$) proxy.php?orig_url=$1

And this is the code for proxy.php:

<?

    include ('simple_cloak_v2.inc.php');
    // should we deny access?
    if (SimpleCloakV2::metaRobotsExcludeProxies(false)) {
        header("HTTP/1.0 403 Forbidden");
        echo 'forbidden … ';
        exit();
    }
    // otherwise echo as it was …

    // construct the original URL
    $url = $_SERVER['SERVER_NAME'] . '/' . $_SERVER['REQUEST_URI'];
    
    // get the contents
    $ch = curl_init();    
    curl_setopt ($ch, CURLOPT_URL, $url);
    curl_setopt ($ch, CURLOPT_HEADER, 1);
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
    $result = curl_exec($ch);
    curl_close($ch);
     
    // do some parsing
    preg_match("#(.*)\r\n\r(.*)#s", $result, $captures);
    $headers = $captures[1];
    $data = $captures[2];
    preg_match_all('#(.*)\r#m', $headers, $captures);
    $split_headers = $captures[1];
    
    // we have to reissue the headers as is
    foreach ($split_headers as $s) {
        header($s);
    }
    
    // echo the body.
    echo $data;
    
?>

More implementation :

1. Add this to all of your headers:

and if you see an attempted hijack…

2. Block the site via .htaccess:

3. Block the IP address of the proxy

4. Do your research and file a spam report with Google.

CMS – SEO Compare , Joomla WordPress Drupal

SEOMOZ have some experiment : here

Within two weeks the WordPress site was so far ahead that it looked as if they competition was over. Slowly however, Drupal and Joomla have recovered until they now lead the pack. The Joomla site now has 5 top ten rankings in every search engine that we're following and the Drupal site continues to climb, only in second place by a small amount. 

Some interesting points.

• Is it a blog? If so, we'll expect it to be updated regularly
• Is it a corporate website? If so, we can expect it to remain relatively unchanged

Basically, Google is asking you to decide what kind of site you want to be and then to walk the walk. If run a blog that used to be updated every day, but now hasn't seen fresh content two months, why should Google rank you? Why should searchers have to wade through months-old blog posts?

What is the one of the easiest ways Google can determine what kind of site you're running? Your software. If you install a WordPress site you are telling Google that you're starting a blog and that it will be updated regularly. If you install Joomla or Drupal, Google has no such expectations. Maybe that is why WordPress started so strongly, but has since fallen behind. If you don't meet expectations, you will be penalised. 

Not just SEO, We should care some topic more,

Usability

The conventional wisdom says that the learning curve for Joomla is much greater than for WordPress.  My opinion is that the conventional wisdom is wise in this case.

The two main things that made WordPress easy for me, someone who built his first website three months ago, are the html editor and the template editor.  With WordPress, you can toggle back and forth between your WYSIWYG and html editors for pages and individual posts.  Additionally, most templates break down the PHP coding into easy to use sections like “header,” “footer,” and “main index.”  This made customizing our site’s theme much easier. 

Bottom Line: Your site will look great in about 5 minutes using WordPress, but if you are willing to trade some extra learning time for a more advanced site, go with Joomla.

Versatility

WordPress makes some effort to help you build the site you want, but Joomla is the clear winner here.

WordPress is built for bloggers, featuring comments, tagging, and virtually everything else your blog needs out of the box. Simple. Easy. Straightforward.

Joomla is the better choice if your site is almost anything but a blog. The tough learning curve is partly a function of the advanced features that Joomla provides. You’ll have to spend some more time with it, and Joomla is obsessed with goofy looking contact forms and FAQ pages, but your non-blogging site will thank you.

For example, it’s very easy once you get the hang of it to extend Joomla with e-commerce applications, banner advertising, and great form builders.

Bottom Line: If your site is only a blog, use WordPress. The more cool-looking static content you need, the more Joomla will help you.

SEO Tools for CMS

Drupal Modules

1. Block Cache creates cached versions of each block displayed on your website. For each cached block only one SQL query is executed thus reducing server load resulting in better performance and faster delivery of content.
2. Find URL Alias is a utility module that lets you search for particular URL aliases. This module requires the core path module which enables search engine friendly URLs for your site. Find URL Alias is especially useful when there are many aliases stored in your database. But keep in mind that changing URLs is something you should avoid.
3. Google Analytics integrates Google's powerful web statistics tracking system with your website. The module enables you to selectively track users by role.
4. Meta tags allows you to define site wide meta tags and specific tags for each piece of content (node). If you use the core taxonomy module (you really should do) meta tags can be assigned automatically by using the terms (tags) you use to categorize your content.
5. Pathauto automatically generates path aliases based on the modules settings. A very powerful module that lets you define different URL patterns based on content types. In the latest version also allows to filter common words. Requires the core path module.
6. RobotsTxt is useful if you run multiple Drupal sites from a single code base and want to have different robots.txt files for each of them.
7. Service links automatically adds links to social bookmarking and blog search services to your content. You can select which services you want to link to, restrict the display based on content (node) types and whether to display links in teaser and/or full page view.
8. URLify automatically generates the path alias for a piece of content based on its title using JavaScript. Requires the core path module. I lightweight alternative to pathauto.
9. XML Sitemap generates an XML sitemap which complies with the sitemaps.org specification. The relative priority of each piece of content is calculated based on content type, number of comments, and promotion to front page. The values of each of these factors can be set in the admin section of the module.

Joomla Modules, Components, Plugins

1. Advanced SEF Bot for Joomla 1.1.x is a plugin that enables search engine friendly URLs. Since Joomla's standard URLs are not really meaningful this one is a must.
2. Dynamic gSitemap is a PHP script that dynamically creates an XML sitemap of your site when GoogleBot visits it.
3. JoomSEO is a plugin that dynamically creates meta tags, changes the title tag on the fly, adds heading tags to content titles and more. Some of the configurable features include: show, hide or override keywords, site name, and content title, adjust element order in the title and heading tag selection.
4. LinX is a link exchange component that lets you manage a reciprocal link directory. Links that are submitted to your site are automatically checked and only added if there is a link back to your site. A large number of links to your website of course has an effect on the PageRank but what really matters are links from quality and trusted websites that have a similar target audience as your site.
5. MetaTags NX creates meta description and keyword tags for your content on the fly. Keywords are generated based on their frequency in the content and stopwords can be excluded.
6. Redirect component lets you redirect old urls to new ones and set their status codes. Remember, changing URLs often is a bad idea.
7. Simple SEO Plugin uses Yahoo to analyze your content and gives immediate feedback on what Yahoo thinks the content is about. Also creates meta tags.
8. Website Validators Tool contains links to validation and site information services that help you make your website more standards compliant, see how you rank who links to you and more.

Joomla SEO

At the suggestion of Bookworm SEO, I decided to write a post about SEO for Joomla sites. [Note: I'm not here to suggest that Joomla is better than WordPress, Drupal, or any other CMS. My goal is simply to discuss some things that can be done with a Joomla site to make it better.] Also, I'm still far from an expert in the SEO field, so take it easy on me.

The big issue with Joomla is that its out-of-the-box settings aren't exactly search-engine friendly. It defaults to creating horrible URLs, its default page title system has issues, and it creates duplicate content in several ways. (Great CMS, no?)

Turning Down the Crap-Factor on Joomla URLs

The default URLs can be improved slightly by turning on "Search Engine Friendly URLs" in the site's Global Configuration menu. They'll go from looking like this:

http://www.yoursite.com/component/option,com_mtree/task,viewlink/link_id,2380/Itemid,35/

to this:

http://www.yoursite.com/content/view/30/55

Better, but still not great, as they're completely lacking in keywords.

The best solution is to simply download and install a plug-in. There are several that get the job done, but the one I use is sh404SEF.

An example of a URL from my site with the above plug-in installed:

http://www.businesstaxbooks.com/index.php/Sole-Proprietor-Tax-Guide/Self-Employment-Tax-Explained.html

Looks better to me. [Note: Here's a chance for somebody who actually knows their stuff to show me up and explain how to get the"/index.php" out of there.]

Solving Duplicate Content Issues with Joomla

The above URL plug-in will fix a great deal of the problems that Joomla creates by pointing so many URLs at the same piece of content. Another important step to take [at the risk of stating the overly-obvious] is to be sure to turn off the "email icon," "print icon," and "pdf icon" in the Global Configuration menu.

Improving Joomla Page Titles

By default, Joomla page titles follow this structure: "SiteName – PageName"

There's been some discussion recently about whether or not to include the site name in the page title. Regardless, I think we can at least agree that it's a good thing to have control over whether or not to include the site title. (And further, to have control over whether it comes before or after the page's title.)

For instance, my site's name (Business Tax Books) is rather long. As such, my longer titles were getting cut off due to character limits. For example, my C-Corp vs S-Corp vs LLC page only had approximately half of its title showing in the SERPS. (And it was the relevant keywords getting cut off, because they came after the site name.)

A quick change in the joomla.php file (located at /includes/joomla.php) will fix this.

Change this:

$this->_head['title'] = $title ? $GLOBALS['mosConfig_sitename'] . ' – '. $title : $GLOBALS['mosConfig_sitename'];

To this:

$this->_head['title'] = $title && $title != "Home" ? $title : $GLOBALS['mosConfig_sitename'];

I take no credit for the above fix. I found it on the Joomla forums. After I made the change, I noticed a substantial increase in search traffic to the page I mentioned above, as well as a few others with long titles. (And that's without even doing anything to rank better!)

WordPress Plugins

1. Add to Any adds links to a large number of social bookmarking sites to your posts.
2. GeneralStats is a statistics components that counts the number of users, categories, posts, comments, pages, words in posts, words in comments and words in pages. Useful for doing keyword research.
3. Google Sitemap Generator creates an XML sitemap of your website. In the current version homepage, posts, static pages, categories and archives are supported. Priority is automatically assigned based on the number of comments.
4. Gregarious is a social bookmarking plugin for Digg, Reddit and Feedburner with update checks via AJAX.
5. Popularity Contest is a counter for posts, categories, archive views, comments, trackbacks, etc. to determine the most popular pages of your site.
6. Technorati Tagging Plugin adds Techorati tags to your posts and enables you to display a tag cloud.
7. WP-Cache is a page caching system to improve your websites performance. Cached pages are stored as static files, reducing server load thus making your site faster and more responsive.
8. X-Valid attempts to convert posts and comments to valid XHTML. Read more on the benefits of Web Standards Compliance.
9.  

New Features

• Truer Edge technology in Photoshop CS5 Extended offers improved edge detecting technology and masking results in less time. Photoshop CS5 Extended also lets users remove an image element and immediately replace the missing pixels with Content-Aware Fill.
• InDesign CS5 powers the transition to digital publishing with new interactive documents and enhanced eReader device support.
• Native 64-bit support in Photoshop, Adobe Premiere Pro and After Effects enables customers to work more fluidly on high resolution projects.
• New Text Layout Framework in Flash Professional CS5 provides professional-level typography capabilities with functions like kerning, ligatures, tracking, leading, threaded text block and multiple columns.
• New stroke options in Illustrator CS5 allow users to create strokes of variable widths and precisely adjust the width at any point along the stroke.
• The NVIDIA® GPU accelerated Adobe Mercury Playback Engine allows Adobe Premiere Pro CS5 users to open projects faster, refine effects-rich HD sequences in real time and play back complex projects without rendering.
• The new Roto Brush tool in After Effects helps users save time by isolating moving foreground elements in a fraction of the normal time.
• Dreamweaver CS5 now supports popular content management systems Drupal, Joomla! and WordPress, allowing designers to get accurate views of dynamic Web content from within Dreamweaver.

Pricing and Availability

Adobe Creative Suite 5 and its associated point products are scheduled to ship within 30 days, with availability through Adobe Authorized Resellers, the Adobe Store and Adobe Direct Sales.

Estimated street price for:

• Adobe Creative Suite 5 Master Collection is expected to be US$2599
• Adobe Creative Suite 5 Design Premium is expected to be US$1899
• Adobe Creative Suite 5 Design Standard is expected to be US$1299
• Adobe Creative Suite 5 Web Premium is expected to be US$1799
• Adobe Creative Suite 5 Production Premium is expected to be US$1699

httpv://www.youtube.com/watch?v=v69S22ZBBqA