Well that was a FUN experience upgrading one of my clients Magento stores from version 1.6 to 1.9.1.1. Magento is one of those platforms that seems to instil the fear of dread when upgrading core functionality and it working as intended.
Upgrading core versions generally is never straight forward
Never have I had a smooth upgrade experience with Magento CE (community edition), not like WordPress, which for the most part is a breeze.
Part of the problem I have found, is that any upgrade issues never show their ugly heads at the point of being upgraded, and only later on they peek up above the surface.
In this particular case, the upgrade process appeared to be super smooth! No problems at all, and the console reported that everything had been upgraded successfully. I sat back thinking, wow that was easy.. what on earth was I worrying about.
Days later PayPal sent an IPN notification email
My heart sunk, when a couple of days later my client contacted me asking about their order statuses not being updated and that they received this unusual email from PayPal. With the subject:
PayPal Instant Payment Notification Warning
Dear {customer name},
Please check your server that handles PayPal Instant Payment Notifications (IPN). Instant Payment Notifications sent to the following URL(s) are failing:
http://www.clientdomain.co.uk/paypal/ipn/
If you do not recognize this URL, you may be using a service provider that is using IPN on your behalf. Please contact your service provider with the above information. If this problem continues, IPNs may be disabled for your account.
Thank you for your prompt attention to this issue.
Yours sincerely,
PayPal
Got out my 12 bore Magento bug hunter rifle & went hunting!
After searching all the error logs in the var/log folder, the website root and the server logs, there was nothing to find! Nothing relating to PayPal IPN, any Curl communication errors, 503’s, 500’s, nothing!
A frustrated hunter I was, with a fully loaded gun and no trail to follow.
I needed help tracking down that dark crevice when this bug was lurking
After contacting PayPal technical support team and WebHostingUK my hosting provider for additional support, I got the usual back-n-forth tennis match.
PayPal kept reporting the server was issuing a 503 (down for maintenance) and my host reassured me, that the server was responding properly to Curl requests and they could find nothing in the logs.
Unless I narrowed down the cause of this issue, I knew we were never going to make any headway and solve this problem.
Just what did PayPals IPN Simulator have to say…
Honestly, not much! I had hopped on to PayPal’s IPN Simulator, entered the clients IPN address and submitted some test data. Each time it returned an elusive HTTP error, which as you can see below does not help!
So I needed to know what was going on and went digging through Magento’s core files, specifically:
- app/code/core/mage/paypal/
- model/ipn.php
- controllers/ipncontroller.php
- lib/Varien/Http/Adapter/Curl.php
Everything appeared to look ok, when compared to the recently downloaded source files, I even added in additional lines of code at each stage of the logic to output the data to custom logs:
Mage::log(“PayPal IPN POST data (my error log): “.print_r($data, true).”.”, null, “my_paypal_ipn_error.log”);
So we were receiving the POST data from PayPals IPN Simulator in the custom log. At this stage our Magento listener should send the unaltered data back to PayPal for verification, which should reply with a single word “VERIFIED” or “INVALID”.
I was sure at this stage that the listener was not sending back the data to PayPal. However, they continually reported that when they manually sent Curl commands to our IPN address, all they would receive were 503 and 500 error codes from our server.
Hours of searching on-line, starting to get desperate I tried the following:
Ok, so here we go.. I found several really interesting blog posts on the subject, with some interesting ideas and tried all of them!
- PayPay IPN probelms – Possibly due to Compilation
Nope, I was not using Magento’s Compiler and the one we were using, I had disabled prior. - Intermittent PayPal IPN failure in Magento – Edmonds Commerce Dev Blog
This made no difference either. - PayPal IPN Integration – SSL Connection Reset By Peer
I looked into adding additional headers in my both the existing response and setting up my own listener/response. Though ideally, I wanted to fix the current installation (as it should work). - Debugging PayPal IPN postback failures in Magento
This was an extremely useful post in understanding the process, but resulted in a similar solution to option 2 above. - Orders Stuck in Pending Payment in Magento Due to Paypal Disabling IPN
I found this article really useful creating a centralised IPN listener and it was working with the IPN simulator, sending/receiving a response. However, it did not solve the problem.
Also…
- I even tried (despite my doubt that) adding all of PayPals IP addresses (there are a lot of them) to my firewall and outbound ACF, just in-case their response was being blocked by the server. Nope.. still nothing!
None of the suggestions made on each of these blogs actually worked in this case, whilst I found them all very interesting and indeed helpful in my education on how the IPN listener works.
SPLAT! Oh that felt good killing that Magento bug.
Ok, if you’re not bored already and have continued reading, I managed to solve the problem without support from PayPal or my web host. Whilst they were both being very helpful and asking for information from each other on things they could try, we were not making any progress.
I had got to the point of exhaustion, absolutely fed up trying to pinpoint the location of the error. It was gone 8pm and my lovely wife had made dinner, whilst doing her best to help me unwind. I decided to simply overwrite all the files via FTP in the folder: app/code/core/mage/paypal/ from the original source and come back and work on it later.
Only an hour later, whilst watching a film the IPN notifications were coming through… YAY!
So being honest, it was more of a blind spray and pray, a last ditched attempt to kill this bug… and SPLAT it went! I can only deduce that either there was a corrupted or missing file, as part of the incomplete process.
In hindsight, I wish I had tried this sooner, but you live and learn.
So others can benefit, I debriefed PayPal and WebHostingUK
Not that I imagine my particular problem is very common, I went to PayPal and WebHostingUK with a debrief of sorts, on how the issue was resolved. I asked PayPal to pass on my ticket to eBay/Magento for future reference.
But I wanted to update them both to mysterious 503 error, that it turns out was part of the response generated in the code here:
- app\code\core\Mage\Paypal\controllers\ipncontroller.php
- app\code\core\Mage\Paypal\Model\ipn.php
There’s a line of code that sends a 503 response if there is an exception when receiving the data:[/vc_column_text][vc_message]Mage::app()->getResponse() ->setHeader(‘HTTP/1.1′,’503 Service Unavailable’) ->sendResponse();[/vc_message][vc_column_text]If they had known this, it might have reduced the amount of back-n-forth emails and helped get to the root of the problem sooner. Being that it was not the server reporting the 503 per-say, but the code itself.
Thanks for reading!
