Category Archives: DevOps

Request Loop

It’s been a while since I last posted…

There is always a reason for that. My reason was a sum of many different variables. Just as the great mentor said, luck is the sum of many coincidences, that’s what happened in my case as well.

Where do I begin?

Jobwise: Capital controls, working day and night, a lot to do and no time to do it…

Blogwise: I had a very strange setup with my blog (and a very very outdated one I might add). Since I am using Heroku, they decided to change their stack and migrate from Cedar 10 to Cedar 14 . Ok I said what the hell lets do it.

Alas, I had a serious problem with libssl0.98 which was built inside my php module and was not supported in Cedar 14. (whoever wants to do the upgrade have a look here first).

Long story short I fixed it, and I also found that many posts I did with various hacks for the pg4wp plugin were incorporated into a single release from kevinoid : here

I will contribute also into some changes that have to be taken into account since the module is quite old and I have previously stated that it’s not at all well written.

That’s not the main point of this post though.

I wanted to share an experience I keep coming across lately.

Now according to popular trends we are experiencing (and will experience in the future) a huge bloom of the microserviced architecture. This guy here explains how and why they decided to go for the microserviced architecture.

I agree. There are many benefits when having a monolithic single (and obsolete at times) repo for web applications. It is a nice solution when your company is scaling, and you have to maintain a lot of different parts. Especially if you have different teams and each team wants to “do their own thing” about a solution.

However it’s not the solution to Everything!

I will elaborate more:

I recently had to debug an http step based procedure (client requests this page, books this ticket, goes there, etc.) that was using 3 different instances of different technologies over http. The one was python and wsgi, the second was php with apache and the third one was ruby with unicorn.

Try to debug this. I dare you. Seriously. I had in my local setup all 3 different instances running with 3 different IDE’s and all running their debugger. Ok, ok you say that Docker will simplify the installation. I agree it does, but it does not help the debugging at all.

The most important thing though isn’t the debug/testing of many different apps over http.

It’s the HTTP by itself.

And believe me, I have seen a lot of “Senior” Devs falling into the same trap of API’zation and doing over and over the same architectural error.

The Request Loop

You won't guess how many time's I've seen this happening...
You won’t guess how many time’s I’ve seen this happening…

Consider the following diagram:

This is the actual loop - when one request is still open, another comes along, and things get messy...
This is the actual loop – when one request is still open, another comes along, and things get messy…

The Browser  sends a request to the Frontend app. Now the Frontend App could forward it (or change it a bit) to the Backend App.

In our setup the backend app was a PHP app.

Now since PHP by default does not support threading (not pthreads), each HTTP request is a different PHP thread, served via apache.

This is very complicating, since you keep a connection (process) open and you open another one which could (at some point maybe) rely on data from the first one. You cannot access that data in between processes.

Not to mention that, you can not either debug this thing, since you insert a break point in the first request procedure, and the second request (which happens a few ms after) is being served without the debugging stopping at that point.

My point is that when you decide to go Microservice’d

Try to avoid request looping, when you need to do something that is synchronous. Or, use something different. Do threading. Use a message queue, or something else.

You will be surprised how much time you will spend trying to debug and understand what is wrong in this set-up.

I will close with the following meme:

Some people, when confronted with a problem, think, “I know, I’ll use threads,” and then two they hav erpoblesms.

Upgrading to WordPress 3.9 in Heroku

As you may already know, our blog is using the WordPress Heroku PHP Buildpack.

Now, since April of 2014 Heroku has updated it’s buildpack to support officially PHP and their final version here. They have done a really fine job, compared to the old legacy buildpack which involved a custom build batch file, and a plain empty php file to procreate the Heroku Dyno to run the Apache process with php process. More details from this awesome guy, here : https://github.com/mhoofman/wordpress-heroku and the dyno changes here : https://github.com/xyu/wordpress-heroku .

Our current installation though uses also WordPress.

Since then WordPress has released some considerable updates, and we need to update to the newest version. So, doing the necessary process, downloading WP3.9.2 , extracting it to the current setup, committing, pushing to heroku master.

At first heroku fails. Since by default it uses the new buildpack and requires composer.json to exist, so as to run Composer and configure afterwards the PHP-Heroku-Buildpack. Now, that was my point of failure since I did not notice that you can keep the legacy buildpack (but hey, new is always better right?) and I agreed to take on the new one.

After following the guide here Getting Started With PHP ,  created a composer.json (currently empty since we did not require anything new) and modified Procfile since we needed to explicitly tell Heroku that we had a custom Apache configuration.

A custom Apache Configuration?

Aye. While on the old buildpack you could initialize the stack with php, some WordPress plugins (such as Jetpack, or Zip support) did multiple requests on the server (I assume Ajax ?) and Heroku did not support concurrent requests in the old buildpack. Zip required to include mod_deflate.so. So I found out this guy: https://github.com/xyu/ who wrote a custom script doing nothing more complex than including to the Apache conf his configuration which in turn enabled the concurrent requests. This file was being loaded at startup, when heroku dyno was running.

Searching the Heroku documentation

..and I found out that if you want to include a http.conf  file you have to do it somehow like this :
web: vendor/bin/heroku-php-apache2 -C apache_app.conf
by modifying the Procfile and inserting the custom config file.

Obviously that did not work.

There were warnings that stated that we could not declare ServerLimit inside the VirtualHost section. Which later on made more sense when I saw how New Heroku buildpack integrates apache…

Aside from that, in the Heroku Logs :

There were enough warnings (and a fatal) like these :
2014-04-30T17:59:48.184463+00:00 app[web.1]: [30-Apr-2014 17:59:47] WARNING: [pool www] child 48 said into stderr: "NOTICE: PHP message: PHP Warning: pg_query(): Query failed: ERROR: missing FROM-clause entry for table "session""
2014-04-30T17:59:48.184466+00:00 app[web.1]: [30-Apr-2014 17:59:47] WARNING: [pool www] child 48 said into stderr: "LINE 1: SELECT @@SESSION.sql_mode"
2014-04-30T17:59:48.184468+00:00 app[web.1]: [30-Apr-2014 17:59:47] WARNING: [pool www] child 48 said into stderr: " ^ in /app/wp-content/pg4wp/driver_pgsql.php on line 140"

I’m not a Postgres expert, I’m more familiar with MySQL so, I decided that I should install my blog locally.

Downloaded the Postgres dump, restored in local instance, along with the new buildpack, ran an apache instance, and voila another error:
LINE 1: SELECT @@SESSION.sql_mode
^ in /var/sites/example.com/www/wp-content/pg4wp/driver_pgsql.php on line 133
PHP message: PHP Fatal error: Call to undefined function wpsql_errno() in /var/sites/example.com/www/wp-content/pg4wp/core.php(32) : eval()'d code on line 1531"

That one was mitigated from these guys here : https://vitoriodelage.wordpress.com/2014/06/06/add-missing-wpsql_errno-in-pg4wp-plugin/

When I was sure that everything works as intended, I finally pushed again.

Nothing. 0. Actually 500:

500

Checked the logs and I was getting a VERY  descriptive message :
2014-22-11T03:17:49+00:00 heroku[router]: Error H12 (Request timeout) -> GET / dyno=web.1 queue= wait= service=30000ms status=503 bytes=0

This error was not helping at all. No matter what I tried after that, I kept receiving timeout, as if the php apache process was eating up all the resources of the dyno. Clearly something is very wrong…

Finally, I decided to use the standard legacy buildpack

And reverted everything, after upgrading to the new WordPress, migrating the database locally and updating the cloud afterwards.
I used the legacy buildpack by running in my local heroku env:
heroku config:set "BUILDPACK_URL=https://github.com/heroku/heroku-buildpack-php.git#legacy"

 

I have just recently found out that the guy who wrote the plugin connecting to Postgres (this guy: https://wordpress.org/plugins/postgresql-for-wordpress/ ) has recently stoped developing it. And later, (although I still have second thoughts on How hhvm is compatible with php 5.3 and more specifically 5.3 in WordPress) Xiao who migrated his blog to HH-VM/NginX/MySQL heroku build pack : https://github.com/xyu/heroku-wp .

As soon as I have updated I will gather results to see how it went.

EDIT: It looks like there is an update for PG4WP I have been missing. Trying it and letting you know…
EDIT2: Nothing. The process keeps hanging by returning :
←[0m at=error code=H12 desc="Request timeout" method=GET path="/" host=www.must-feed.com request_id=fcaa61c9-1bbf-48d2-9b3c-797ca14ffa58 fwd="192.168.84.223" dyno=web.1 connect=1ms service=30000ms status=503 bytes=1240