How Mouseflow Helped Me Find Invalid Google Adwords Visitors, Bots!

mouseflow-heatmapI was reading an article about SEO a few weeks ago and the writer mentioned that you MUST use some kind of heat-map or mouse tracking to see what your visitors are up to. I have heard about this type of thing but dismissed it as irrelevant for my purposes as I don’t spend much time on SEO anyway, I forget why I was reading the article in the first place. He mentioned in the article that mouseflow.com is what he recommends as they have a free tier with limited functionality. My interest was piqued and I checked out mouseflow. I was sold in about 30 seconds as they let you test what you’ll see by getting you to interact with one of their pages and then playing your interaction back to you. If you haven’t already head over to mouseflow and do the demo then keep reading.

After putting the tracking code into my website and watching a few visitors I decided I could use this to see how visitors from different channels (sources) interacted with my site. I had stopped using Google Adwords a few weeks ealier because of the terrible conversion rate. About 80% of visitors interacted with one page only and left straight away (bounced). I suspected they were bots or some automated way of increasing Google Adsense advertises revenue. As you probably know Google sells adverts to businesses (Adwords) and buys advertising space on websites (Adsense). I have been using both on and off for about 10 years with extremely poor results.



With my new mouseflow account I was able to see each visitor that came to the site, all the details about where they came from and what technology they were using and most important of all, what they actually did on the site. I turned my Googlge Adwords campaign back on and allocated $35 for testing over 5 days. The $35 provided me with 641 clicks to investigate in mouseflow.

Detecting Patterns

I was able to pick up patterns very rapidly with the information in mouseflow. I picked up two interesting trends, one particularly interesting indeed. I was receiving a lot of visitors from Myanmar (Myanmese??) who were all using android mobile devices. Most of them were bouncing after scrolling part way down the first page, but many of them displayed normal behavior and clicked through a few pages. It wasn’t until i watched them a few times each that i picked up an interesting similarity. *Scroll*… *scroll*… *wait*… *click*. The click was usually the first hyperlink in the content of the first page, which is a sign-up button. This is normal… then on the sign-up page the same scroll, scroll, wait, click happened. This time the closest thing to click was a submit button on the sign-up form. But the visitor would never enter any information in the sign-up form, so it would display itself again with an error. Several of the visitors would then scroll, scroll, wait, click on the page with the error, yep you guessed it, the form was submitted again without any info! See the image below for a sample of my Myanmese visitors.

mouseflow-recordings


Now I am very sure that the majority of my Google Adwords visitors (90%+) are bots. The ones that bounced were almost certainly bots but now there is a percentage, a large percentage of visitors that interacted with two or more pages that are definitely bots also! At this point I sat back and had a small epiphany. Google buys and sells Ad space and takes a cut, just a middle-man. Why would they have an incentive to reduce this type of fraudulent activity? Well… they wouldn’t. I did some reading and some sites that claim to be in the know believe that around 50% of all internet traffic is non human. This is made up of of good bots and bad bots. I believe I’m paying for bad bots to visit my sites through Google!

The Resolution

Update! (27/11/2018) Google gave me my money back!

— Google Response —

 Hi Tony,

Thank you for your patience while we reviewed the details to find our the invalid activity on your account.

Post the review is complete, I would like to share that we noticed some clicks and impressions that didn’t fit the profile of normal user traffic.

We verified that our monitoring software has already filtered out some of this activity, and have confirmed that you weren’t charged for clicks that were automatically discarded. You can find data on the invalid clicks that have been automatically filtered from your account in the Campaigns tab in your account. For more information, please see: https://support.google.com/adwords/bin/answer.py?answer=44008.

In addition, please know that we have already applied a credit to your account for potentially invalid clicks that weren’t automatically filtered from your reports during this period. This credit is a result of our offline analysis and will be visible on your Billing Summary page in the next 3 to 4 weeks.

The security of AdWords advertisers is a top priority for Google. Our proprietary technology analyzes clicks and impressions in an attempt to determine whether they fit a pattern of use intended to artificially drive up an advertiser’s clicks or impressions, or a publisher’s earnings. Please be assured that we will continue to monitor all clicks on your ads to prevent abuse. For more information about the steps we take to combat invalid click activity, please visit http://www.google.com/ads/adtrafficquality/invalid-click-protection.html.

We appreciate your business with us and please do not hesitate to share with us if there is anything you think we can do to better your experience with us. If you need further clarification, all you have to do is respond to this email and I will waste no time in getting back to you.

Best,

Pawan
Google Inc. | The Google AdWords Team | 1-866-2-GOOGLE | adwords.google.com

— End —

IBM Bluemix with Watson is Awesome

IBM WatsonI am always interested in the high availability could services offered by Google (google cloud), Microsoft (Azure), Amazon (AWS) and IBM (Bluemix). I checked all of them out and tested some of the different features of each recently for a project. I eventually settled on AWS based on pricing, flexibility and performance. But, the most interesting feature I came across was on IBM’s Bluemix platform.


There are several ways you can tap into IBM’s AI, Watson. You can harness the power of this AI to use in your website or web applications. I was skeptical at first, assuming it would be garbage, but I was wrong. There is something called ‘personality insights’. Where you enter a bunch of text written by someone and Watson will attempt to tell you about the personality of that person based on that text. I copied and pasted some text from an old university assignment, which was the first thing i could find. It pinpointed a couple of things about me that I would agree with! Try it yourself console.ng.bluemix.net/catalog/services/personality-insights


Personality profile

 

So I decided to explore more. There are a heap of different services that Watson offers such as interactive chat / help for users on your website, translation, tone and emotion analysis and visual recognition tasks. There is a full list available here www.ibm.com/cloud-computing/bluemix/watson. My problem when I last tested the services was that the API was only accessible from Node.js, and I know nothing about Node.js, but a lot about php. After giving up on my second Node.js tutorial (it just wasn’t sinking in) I threw the towel in. But now I see they are offering their services in CURL, which I know php supports!


I think this is a fantastic service and the other major cloud service providers are going to try and offer their own alternative, some already are. But IBM have a head start as they have been developing Watson for a while now. I’m sure you’ve read about Microsoft’s recent failure, Tay, who lasted 16 hours before being corrupted into asking people for sex and becoming a fascist en.wikipedia.org/wiki/Tay_(bot).

How to repair a damaged or corrupt PDF not starting with %PDF-

Just this morning I received a PDF from an international supplier, which we needed urgently. Because of the time difference they were asleep when we discovered there was a problem with the PDF. We needed a document to clear some goods arriving at a port. If you search online there are a myriad of programs that can fix your PDF’s online or offline, but most of them cost money or leave some nasty watermark on your PDF. We needed the original intact without modification and I didn’t think I should have pay $50 for the privilege.



The Error – Acrobat could not open xxxxxx.pdf becasue it is either not a supported file type or because the file has been damanged…

acrobat-error

So what did I do? These are the steps that worked for me.

Spoiler – My PDF file didn’t start with %PDF-. This will not work if you have a different problem!

1. Open the PDF with Notepad or Wordpad

On windows you can use either notepad or wordpad to open any file to see the contents. Beware if it is large you may be waiting a while! To do this find the file in windows explorer or save it on your desktop while you are working with it. Right click and choose “Open with”. You then click on “Choose default program”. Make sure you untick ‘Always use the selected program…’ so you don’t associate all your PDF’s with a text viewer!

choose-default-program

2. Search for %PDF-

All PDF documents should start with %PDF- to be valid. Sometimes the software creating the PDF doesn’t do this but the rest of the PDF is valid and just needs to be modified a bit. After looking at the start of my document I could immediately see the problem. The PDF included part of the email inside the PDF file, in plain text. The %PDF- marker was way down the file after a bunch of stuff. Apparently this should be in the first 1024 bytes of the file to be valid.

pdf-in-wordpad

search-for-%pdf

Based on this I decided that I would manually remove all the garbage before the %PDF- marker and save the file. To do this properly you have to use a binary safe program as all the special characters you see in the PDF is binary data that will be further corrupted if you edit it with notepad or wordpad.

3. Download a Fee Hex Editor

A hex editor is a program that allows you to see the contents of the file and change it without corrupting any of the other binary data in the file. We could have gone straight to this point, but, if your file started with %PDF- when viewed with notepad or wordpad, there is no need for a hex editor. There is something else wrong with your PDF!

I downloaded Free Hex Editor Neo. Installed it and opened my troublesome PDF.



4. Search for %PDF- and delete all the garbage

Once you have the PDF open you should see a bunch of hexadecimal numbers on the left and a narrow column of characters on the right. From here you can search for %PDF- by pressing Ctrl + F. You should select everything before the % all the way to the top of the file and then press delete. Once you are finished the first character that appears in the right column should be the % in %PDF-. See the series of images below for the steps I took with my PDF.

hex-editor-neo-01

hex-editor-neo-02

hex-editor-neo-03

hex-editor-neo-04

5. Save file and Test Opening it

Save the file in your hex editor, which will write your changes to the file.

Then cross your fingers and open the file up with your usual PDF viewing program. If all went well you should see what I saw, a lovely PDF with it’s data intact.



fixed-pdf

How to share Kindle Books

Share Kindle BooksSo you’ve made the switch to a Kindle, well done. Now for the most part it is all good. But when you really enjoy a book and want to share or have finished a book and want to pass it on, it’s a little trickier. This was super easy with physical books. You basically grabbed your book, walked over to your friend / family member / complete stranger and handed them the book, job done. Now you need to get a bit more technical to share eBooks.



1) Lend or Borrow Kindle Books from Amazon

From the Amazon website you can loan a book for up to 14 days. I believe you can only do this once. And while it is on loan you cannot read it yourself. How thoughtful of Amazon to simulate the physical book experience so accurately.

For full details Click Here

No everything can be loaned, so if the option isn’t there it can’t be loaned. If you are in Australia like me I don’t think you can loan anything at all. None of the books in my kindle library are available to loan. This is either a fairly large coincidence or I live in the wrong part of the world to loan Kindle books.



2) Set up Family Library

If you want to share books in your household between two adults and four children, Family Library is for you. Theoretically you could probably trick Amazon into thinking your four closest friends are your children but their accounts would have to bet set up like that from the start, so you would have to be playing the long game already.

For full details  Click Here

Once again my location on the globe seems to exclude me from possibly having a family that should share Kindle books.

3) Make Your Device Part of Someone Else’s Account

Like the previous point but not quite as complex, you would have to trick Amazon into thinking your device is actually someone else’s device and add it to their account. You could in essence share the one Amazon account with multiple devices to freely send the ebooks between them. However, you’d want to be pretty comfortable with each other as it would mean sharing an entire Amazon account, things like this might show up in your purchase history – Pussy Snorkel (check out the top review).



4) Calibre eBook Management

  • Download this – Calibre eBook Management
  • Plug in your kindle, ipad, tablet or phone. (for android set your usb mode to media transfer)
  • Copy all your books to the Calibre library on your computer.
  • Plug in ‘your other’ kindle, ipad, tablet or phone.
  • Copy all your books from your library to the other device.

From the steps above and using the Calibre help you should be able to see how you can share you books freely between ‘your devices’.

Contact me if you need any help and I’ll add some further instructions.