Andrew Carter

Dealing with Needlephobia Using Machine Learning

Tue, 21 Dec 2021 00:00:00 +0000

For the last year and a bit, news websites around the world have been competing with each other to see how many photos they can host of people with needles sticking out of their arms. As a needlephobe, this invasion of my safe space has not gone unnoticed. I had previously thought that the news cycle might soon pass, but I realised recently that we will be talking about vaccinations and boosters for a while yet.

This left me with two options:

Face my fears, confront my needlephobia and work towards being able to rationally handle the concept of an injection
Find a way to hide images of needles whilst I am browsing the internet so that I can continue living as before without self improvement

Naturally, the second option was far more appealing.

The Plan

A good first step for any software project is to break the problem that you are trying to solve down into smaller problems. In this case, there were two clear problems that required solving.

The first problem was that I needed some software that could look at an image and detect whether it contains a needle. This is an image classification problem, and these problems used to be very difficult to solve. However, recent advances in the field of machine learning have made creating and using image classification models quite easy.

The second problem was that I needed some software that could find, test and hide images of needles that I encounter on the internet. The BBC news website is by far the most common source of offending images in my life, so I considered making a proxy website for my own personal use (I even came up with a name: Needle Free BBC). This would have the advantage of being easily available on all of my devices but the disadvantage of limiting the operational scope to a single website. I ended up deciding that a browser plugin was a more complete solution.

Building the Model

The first thing that I needed to gather was training data. If I was going to teach my model to guess whether an image contained a needle, that meant that I needed lots of images of needles (I also needed lots of images of things that are not needles). The more images I had, the more accurate my model was going to be.

The more astute readers may have already noticed my predicament.

The call was marginal at best, but I decided that it would be preferable to consolidate all of my needle viewing into a few hours. The alternative was living with the prospect of frequently encountering such images for months and maybe years to come.

Initially it was difficult (and slow) to find enough good training examples for my dataset. The BBC website hosts an offensive number of needle photos, but not the hundreds that I needed to train my model. Luckily, they cite their supplier.

If the BBC website is a crack den for needle photos, then Getty Images is the coca plantation. When I was done my combined dataset (training and validation) contained 966 images and 548 of those were of needles. Doing this gathering made me feel pretty sick, and it is safe to say that I am glad this part of the project is behind me.

It is worth mentioning that my definition for whether or not an image contained a needle was pretty broad and I included images of people receiving vaccinations where the needle was not particularly visible. These images actually bother me the most, so the dataset contained lots of them.

I also cheated a bit when building my model. Training a good image classification model from scratch requires a huge amount of data. You can get a head start by using a pre-trained image classification model and then adding some extra layers on the end which you train against your own dataset. A reasonable mental model of this is that the pre-trained image classification model already knows how to see, and I added a bit on the end that taught it how to recognise needles.

The Keras machine learning API is so good that it is trivial and expressive to define this model in Python:

import tensorflow_hub as hub
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout

# We need to resize all images to be 224x224 pixels. The additional dimension of size 3 represents the three colour
# channels (red, green and blue). These dimensions are fixed by the MobileNetV2 model that we are using for our base.
IMAGE_SHAPE = (224, 224, 3)

mobilenet_v2 = 'https://tfhub.dev/google/tf2-preview/mobilenet_v2/feature_vector/4'

model = Sequential([
    # We do not want our base layer to be trainable (because it has already been trained and we do not want to overfit).
    hub.KerasLayer(mobilenet_v2, input_shape=IMAGE_SHAPE, trainable=False),
    # Dropout layers help with overfitting by randomly hiding input units during training (preventing the model from
    # relying heavily on individual neurons).
    Dropout(0.4),
    Dense(1)
])

Building the Plugin

Google Chrome extensions must include a manifest file that describes how the extension should be loaded. Here is mine:

{
  "manifest_version": 2,
  "name": "Needlephobe",
  "version": "0.1",
  "description": "Hide pictures of needles",
  "background": {
    "scripts": ["tf.min.js", "needlephobe_background.js"],
    "persistent": true
  },
  "content_scripts": [
    {
      "matches": ["*://*/*"],
      "js": ["needlephobe_foreground.js"]
    }
  ],
  "web_accessible_resources": ["resources/*"],
  "content_security_policy": "script-src 'self' 'unsafe-eval'; object-src 'self'"
}

There are two key parts to my extension, the foreground script and the background script. The foreground script is configured to be injected into every open web page. The background script will run persistently in the background and communicate with the foreground script using messages. This means that the model only needs to be opened once in the background. It also means that pages do not need to be blocked whilst images are examined, as this processing happens in the background.

The foreground script is pretty simple. For each image element that is detected, the URL for this image is sent to the background script. If a response is received saying the image is a needle, then the image is hidden.

function checkImage(image) {
  chrome.runtime.sendMessage({ url: image.src }, function (response) {
    if (response.isNeedle) {
      image.style.filter = "brightness(0)";
    }
  });
}

The background script is a bit more involved. The model is loaded, then the script waits for messages containing URLs from the foreground scripts. Once it has a URL, the image is loaded and processed to fit the input dimensions of the model. Then the image is checked against the model to see if it passes the barrier for being a probable photo of a needle. The script responds to the foreground script with this classification.

async function main() {
  tf.ENV.set("WEBGL_PACK", false);

  await tf.ready();

  model = await tf.loadGraphModel("resources/model.json");

  let cachedUrls = {};

  chrome.runtime.onMessage.addListener(function (
    request,
    sender,
    sendResponse
  ) {
    if (request.url in cachedUrls) {
      sendResponse({ isNeedle: cachedUrls[request.url] });
    } else {
      let imageElement = new Image();

      imageElement.onload = () => {
        tf.tidy(() => {
          let image = tf.browser.fromPixels(imageElement);
          let resizedImage = tf.image
            .resizeBilinear(image, [224, 224])
            .toFloat()
            .div(tf.scalar(255))
            .reshape([224, 224, 3])
            .expandDims(0);

          let prediction = model.predict(resizedImage);
          let isNeedle = prediction.dataSync()[0] < -2;
          cachedUrls[request.url] = isNeedle;
          sendResponse({ isNeedle: isNeedle });
        });
      };

      imageElement.src = request.url;
    }

    return true;
  });
}

Summary

Initially, the plugin generated a few false positives. I switched the behaviour so that instead of hiding photos it added a red or a green border depending on the classification. This was valuable because it made it easy to see the types of images that were confusing the model. I could then expand the training dataset to include more difficult training data, which greatly improved the results.

I considered making this extension publicly available on the Chrome Web Store, but if you want to do this then you need to pay to register as a developer and you probably need to actually maintain the extension afterwards.

All the code is up on GitHub; so if anyone is motivated they can package it up and distribute it.

In conclusion, if you ever feel like your irrational fears are holding you back in life, just use software engineering to change the world around you so that you do not have to confront them anymore.

My Development Workstation

Sun, 24 May 2020 00:00:00 +0000

I am quite frugal with my spending, and I spend quite a lot of time trying to identify the best value equipment. This post explains my current software development workstation and links to where the equipment I have selected can be purchased.

As I am not a gamer, all of the decisions I have had to make have focused entirely on improving my efficiency developing.

Luckily, I built my computer just before the United Kingdom entered lockdown due to COVID-19. I have noticed that the price of a lot of the items that I purchased creating this workstation has since increased by quite a bit. I imagine that this is because the pandemic has caused a shortage in supply and an increase in demand for many computing components.

Quick Links

Computer
Monitors
Accessories
Audio
Software
Furniture

Computer

The processor that I selected was the AMD Ryzen 5 3600. Multiple CPU benchmarking websites showed that it offered the best performance within my budget range by some margin. As this is not a gaming computer, reliability is more of a concern than overclocking and the stock CPU cooler is more than sufficient.

I spent a bit of time trying to establish whether the B450 TOMAHAWK MAX motherboard was compatible with the CPU that I had selected. There was some chatter on forums about have to update the BIOS of the motherboard to get it to work. I did not have to do this, and the system posted and booted first time without issue.

When selecting an SSD, I nearly got caught out buying one with a SATA interface rather than an NVMe interface. SSDs are so fast that they are limited by the SATA bus speed, so you get a big chunk of extra performance for not very much money if you purchase one with an NVMe interface. The SSD that I chose was the WD Blue SN550 500GB NVME.

The RAM kit that I selected was the Termaltake Toughram Z-One RGB 16GB DDR4 3600MHz. I chose 16GB because 32GB felt like overkill and I know that I can always upgrade later if required. It took a bit of time to settle on 3600MHz, but it felt like that was the number that hit the price performance sweet spot I was looking for. I wanted RGB RAM because I knew I was buying a case with a glass panel.

The power supply was probably the component I was least interested in. I ended up selecting the Termaltake Litepower 550W PSU because it was good value and the reviews suggested the cables were good quality.

I would probably class the graphics card that I selected as the only mistake that I made during this build. I chose the Radeon RX 570 because it provided three DisplayPort interfaces and had more memory than the equivalent NVIDIA card that I was considering. The reason I regret this purchase is because I do experiment with personal machine learning projects. When I saw that the AMD ROCm Platform had an official tensorflow upstream package, I had hoped that was an indication of AMD catching up NVIDIA in the field of GPU computing. Unfortunately, the very first model I built crashed for an unexplainable reason when constructing a MaxPooling layer through Keras. If you are at all interested in using your graphics card for machine learning then you should stick with NVIDIA products. The support and stability of the AMD ROCm Platform is still a long way off.

I wanted a case with a glass panel on the side, because I think that these look quite nice. If you are looking for something similar, be careful of cheaper cases that have an acrylic panel. In the end, I chose the EG Black & Mesh ATX Tower Case. I also wanted a case that had two front fans and one rear fan. This creates an outward pressure gradient from inside the case which helps to prevent against dust ingress. Although I would not class it as a mistake, I would probably not recommend this case. The drilling of the holes in the glass panel did not quite match the case, which makes it very easy to cross thread the screws that hold it in place. I have definitely ruined one of them and, although it still bites, it sits at a slight angle and makes me scared every time I have to open the case up.

Monitors

If I had to pick, I would say that my monitors are by far my favourite part of my workstation. As they are primarily used for software development, they needed to have a good screen resolution and lots of space. I chose Samsung UE590D 28” Ultra HD monitors as there were no other 4K monitors of the same quality that were even close to the same price point. I originally purchased three of these, however the three of them together only barely fit on my desk so I decided that the third was unnecessary.

I was prepared to spend a bit more to avoid the cheapest monitor stands that were on the market. I ended up going for the PUTORSEN PC Dual Monitor Arm 17”-32” because they supported 28” monitors (many stop at 27”) and looked like they were going to be stable enough to hold the weight.

It is almost impossible to get the monitors to line up perfectly without any gap between them. I initially tried using some tape on the back to secure them, this worked well but it was a ballache if you ever had to move them. Instead of trying to get them to match perfectly, I have since opted to leave a 1cm gap between the panels. This way it does not bug me if they are 1mm out of alignment with each other.

Accessories

My criteria for a keyboard and mouse was quite simple. Both needed to have a cable, because I cannot afford to have the interruption of changing batteries or having to wait for something to charge. I wanted a mechanical keyboard with RGB backlighting, so I went for the EG Carbon MK II and the EG Nitro Mouse. If I have one complaint, it is that the mouse scroll wheel is a bit squeaky.

Audio

My recording equipment was purchased a few years ago, but it is still worthy of a mention. I have a Behringer C-1 condenser microphone. This requires a mixer that is capable of supplying phantom power and a USB audio interface. I chose the SubZero SZ-MIX04 as my mixer and the Behringer U-Control UCA222 as my USB audio interface. The mixer appealed to me as it could supply phantom power to two microphones, which I thought could be useful for podcasting.

If you are looking to purchase a microphone, I recommend first going to YouTube and checking to see if you can see any reviews of the models that you are interested in. Most reviews will actually include recordings taken with the microphone, so you can hear how the recordings sound for yourself.

Software

My productivity developing software is almost entirely dependent upon a tiling window manager and I get very frustrated if I ever have to work on a machine that does not have one. The window manager I am most familiar with is i3. I am also most familiar with debian based Linux distributions, so Regolith Linux seemed like an obvious choice.

Over time I have evolved my i3 configuration to my preferences, and I am now very happy with it. I find it effortless and fast to arrange my desktop exactly how I want it, and get the most out of my available screen space.

Furniture

My chair is definitely not anything to write home about, and my partner hates it. My desk, however, is something that I am quite proud of. As you can see, I built it to perfectly fit into an alcove in our study. Building it was very easy, I simply attached some table legs from B&Q to a kitchen worktop (also from B&Q) that was cut to size in store. I needed an extra leg in the middle underneath the table to stop it from bowing, but this leg does not bother me and I hardly notice that it is there.

Encrypting Integer Primary Keys in Python

Sun, 17 May 2020 00:00:00 +0000

UUIDs are popular among developers for a number of reasons. These include:

They can be generated by seperate actors with a very low chance of collision
They do not leak information about a system

This article will ignore the first point, and examine the second in more detail.

Sequential Primary Keys

Consider an application that needs to manage a table of orders. With a sequential primary key, the schema for the table might look like this:

CREATE TABLE orders (
    id INT(11) NOT NULL AUTO_INCREMENT,
    # Other fields...
    PRIMARY KEY (id)
);

If we had a REST API route that retrieved these orders, we would see queries such as GET /order/5 and GET /order/167. This leaks unnecessary information about our application to the outside world, it might allow a rival to work out how many orders we had processed and learn confidential information about our business. It also opens up another attack surface, as it makes it easy for an attacker to predict the identifier of past and future orders.

Using a UUID as a Primary Key

Using a UUID as a primary key mitigates some of these problems. Depending on how they are generated, UUIDs can be predictable (although much less so than integer primary keys). However, they do not leak information about table volume. Our REST API would now see queries such as GET /order/123e4567-e89b-42d3-a456-556642440000 and GET /order/cd41294a-afb0-11df-bc9b-00241dd75637. Our table schema might look like this:

CREATE TABLE orders (
    id VARCHAR(36),
    # Other fields...
    PRIMARY KEY (id)
)

If we were using UUID_TO_BIN() and BIN_TO_UUID(), we might be able to store the UUID more efficiently as a binary field:

CREATE TABLE orders (
    id BINARY(16),
    # Other fields...
    PRIMARY KEY (id)
)

Performance Problems

The problem with using a UUID as a primary key is that some database storage engines, such as InnoDB, use an index-organized data storage technique. This means that when inserting randomly distributed values (such as a UUID) the engine has to insert data randomly throughout the table, which is not very efficient. For these storage engines, it is much more performant to use a sequential primary key.

Some application developers solve this problem by using a combination of both fields together:

CREATE TABLE orders (
    id INT(11) NOT NULL AUTO_INCREMENT,
    uuid VARCHAR(36) NOT NULL UNIQUE,
    # Other fields...
    PRIMARY KEY (id)
);

Another option, is to revert back to our original schema design with a simple sequential primary key and use cryptography to encrypt the identifier before it is exported from the application.

Encrypting Integer Primary Keys with Python

As a solution to this problem, I have created a Python class that can be used to encrypt and decrypt primary keys. The class depends on the cryptography module. A demonstration of the API within the Python shell is shown below:

>>> PrimaryKeyEncryptor.generate_secret()
'134810bef265eabbfcc33c8956b9c870'

>>> PrimaryKeyEncryptor('134810bef265eabbfcc33c8956b9c870').encrypt(1)
'ec68e9d12ae27ce7ce9bad7488ecf665'

>>> PrimaryKeyEncryptor('134810bef265eabbfcc33c8956b9c870').encrypt(2)
'd2f225fd542442c6e12ab19943749391'

>>> PrimaryKeyEncryptor('134810bef265eabbfcc33c8956b9c870').decrypt('ec68e9d12ae27ce7ce9bad7488ecf665')
1

>>> PrimaryKeyEncryptor('134810bef265eabbfcc33c8956b9c870').decrypt('d2f225fd542442c6e12ab19943749391')
2

Our REST API would now see queries such as GET /order/ec68e9d12ae27ce7ce9bad7488ecf665 and GET /order/d2f225fd542442c6e12ab19943749391. Another side benefit of this technique, is that it allows identifiers to be validated without touching the database.

In the example below, I change a single bit of the cipher text and the encryptor can recognise that it is not valid:

>>> PrimaryKeyEncryptor('134810bef265eabbfcc33c8956b9c870').decrypt('d2f225fd542442c6e12ab19943749393')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/test.py", line 105, in decrypt
    raise ValueError('The encrypted primary key is invalid')
ValueError: The encrypted primary key is invalid

This works because the block size of the algorithm chosen (AES-128) is 128 bits, but we only have 64 bits to encrypt. This means that we have space to encrypt it twice. When we decrypt the cipher text, if the first 64 bits do not match the last 64 bits we know the cipher text is invalid.

The code for this class is shown below:

"""This module contains a python class that was designed for encrypting sequential integer primary keys

    Typical usage example:

    # Generate a secret (done once)
    secret = PrimaryKeyEncryptor.generate_secret()

    # Instantiate encryptor
    encryptor = PrimaryKeyEncryptor(secret)

    # Encrypt
    encrypted = encryptor.encrypt(primary_key)

    # Decrypt
    decrypted_primary_key = encryptor.decrypt(encrypted)
"""
from os import urandom
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.ciphers import Cipher
from cryptography.hazmat.primitives.ciphers.algorithms import AES as Algorithm
from cryptography.hazmat.primitives.ciphers.modes import ECB as Mode


class PrimaryKeyEncryptor:
    """This class uses AES-128 in ECB mode to generate a secure mapping between a 64 bit integer and a 32 character hex
    string.

    Note:
        In many other contexts, the use of AES-128 in ECB mode is insecure. Please do not use this example for anything
        other than the explit purpose of encrypting sequential integer primary keys generated by a storage engine.
    """

    def __init__(self, secret: str):
        """Initialize a PrimaryKeyEncryptor class.

        Args:
            secret: A 32 character (16 byte) hex string which is used as a key for encryption and decryption.

        Raises:
            ValueError: If the secret is not a 32 byte hex string.
        """

        secret_bytes = bytes.fromhex(secret)

        if len(secret_bytes) != 16:
            raise ValueError('The secret for the PrimaryKeyEncryptor must be 16 bytes in hexadecimal format')

        algorithm = Algorithm(secret_bytes)
        mode = Mode()

        self.cipher = Cipher(algorithm, mode, backend=default_backend())

    @staticmethod
    def generate_secret() -> str:
        """Generate a secret that can be used to instantiate the PrimaryKeyEncryptor class.

        Returns:
            A 32 charachter (16 byte) hex string which is suitable to use as a key for encryption and decryption.
        """

        return urandom(16).hex()

    def encrypt(self, primary_key: int) -> str:
        """Encrypt an integer primary key

        Args:
            primary_key: An integer to encrypt, this cannot be more than 64 bits in size.

        Returns:
            The encrypted integer primary key as a 32 charachter (16 byte) hex string.

        Raises:
            ValueError: If the primary key is not an integer of 64 bits or less.
        """
        primary_key_bytes = primary_key.to_bytes(8, byteorder='big')

        encryptor = self.cipher.encryptor()

        cipher_bytes = encryptor.update(primary_key_bytes * 2) + encryptor.finalize()

        return cipher_bytes.hex()

    def decrypt(self, encrypted_primary_key: str) -> int:
        """Encrypt an integer primary key

        Args:
            encrypted_primary_key: The encrypted integer primary key as a 32 character (16 byte) hex string.

        Returns:
            The decrypted integer primary key.

        Raises:
            ValueError: If the encrypted primary key is not valid.
        """
        cipher_bytes = bytes.fromhex(encrypted_primary_key)

        if len(cipher_bytes) != 16:
            raise ValueError('The encrypted primary key must be 16 bytes in hexadecimal format')

        decryptor = self.cipher.decryptor()

        plain_bytes = decryptor.update(cipher_bytes) + decryptor.finalize()

        if plain_bytes[:8] != plain_bytes[8:]:
            raise ValueError('The encrypted primary key is invalid')

        return int.from_bytes(plain_bytes[:8], byteorder='big')

Ordered UUIDs

When I shared this post on Twitter, somebody pointed out another alternative:

There is also the possibility to rearrange the structure of the uuid to have the time part up front. We have done this with @ramsey /uuidhttps://t.co/qeuWs9jZO4 https://t.co/q0WhjfZGRb
— Andreas 'Ando 🦆' Frömer (@icanhazstring) May 17, 2020

The basic concept is that a UUID is constructed of many different components, including a timestamp. However, the structure of a UUID does not ensure that UUIDs appear in chronological order when they are sorted. By rearranging the UUID such that this is true, it is possible to improve their performance as a primary key quite significantly. A blog post on Percona covers this topic with benchmarks in detail.

Make PHP Great Again

Mon, 25 Jun 2018 20:00:00 +0000

Apologies for the clickbait title, but if you are a PHP developer, I really think that this topic deserves your attention.

PHP has had a love-hate relationship with much of the programming community for a while. An almost complete introduction to the topic of PHP hate can be found in the famous blog post PHP: a fractal of bad design. In this post, the author hilariously cuts down almost every aspect of the PHP language, and sets out a rather convincing argument for consigning PHP to the archives of programming history. The article is now slightly out of date (it was published in 2012), but much of its criticism remains valid.

Personally, I simultaneously love and hate pretty much every language that I have ever been introduced to. I love C and how it taught me low level programming concepts, but I hate how it takes two hours to try and join two strings together. I love Python because of the machine learning and Raspberry Pi communities, but I hate how it is practically impossible to achieve type safety and proper object oriented design. I love JavaScript and the asynchronous programming techniques that it has encouraged, but I hate how I need about 9000 dependencies in my node_modules directory to serve a hello world application on port 80.

There is lots that I love about PHP too. When I was younger, without funding for servers, shared free PHP hosting was all I had. It made object oriented programming accessible to me, and I love the package manager and its ecosystem.

With all that said, I passionaly hate the PHP core functions.

Want to map an array? array_map($callback, $input)
Want to filter an array? array_filter($input, $callback)
Want to get the type of a resource? get_resource_type($resource)
Want to get the type of a variable? gettype($variable)
Want to check if a variable is a string? is_string($variable)
Want to check if a variable is set? isset($variable)

PHP has all of the scars of a language that has been added to, without proper consideration, by many different people over a long period of time. The process for making changes to the language is now much stricter, but undoing the damage that was done is something that needs more discussion.

The alternative, is accepting these glaring inconsistencies as part of the language until it dies. The arguments for fixing or leaving these issues will not change over time, the only difference is that it might be easier to introduce backwards compatibility breaks in the future because more programmers will have gone elsewhere. Every year huge companies such as Google and Microsoft invest significant resources into trying to create a better programming language. If PHP wants to compete in the future with languages that have been designed with the benefit of hindsight, it has to clean out the crap every now and then.

In theory, the process for fixing the PHP core functions is simple. The first step is to introduce deprecations to functions that need to be replaced, and ensure that replacements are available and recommended for use. After a period of time, the deprecated functions that nobody is using any more can be removed.

Good replacements have been discussed before too. There was an RFC for fixing the inconsistent function names that was tabled in 2015 but never made it anywhere. My favourite suggestion, is that introducing scalar objects could open the door for a new API that is inspired by common sense rather than the C POSIX library.

// Rather than
$lowerCaseFruit = ['apples', 'oranges', 'grapes'];

$upperCaseFruit = array_map(function ($fruit) {
    return strtoupper($fruit);
}, $lowerCaseFruit);

// We could have
$lowerCaseFruit = ['apples', 'oranges', 'grapes'];

$upperCaseFruit = $lowerCaseFruit->map(function ($fruit) {
    return $fruit->toUpper();
});

In reality, the process for fixing the PHP core functions is more difficult. The first step is actually convincing the PHP internals group that this cause is worth the effort and the backwards compatibility break. Joining the internals community is a pretty opaque process and it takes time (and goat sacrifices) to accrue the karma and respect necessary to actually action something yourself. But, if the wider community bangs its collective drum loudly enough, there is a chance that some of the key players in the group will set their brilliant minds to this task.

The reason for this post, is that the release of PHP 8 is current being discussed on the internals mailing list. The current suggestion is that PHP 7.3 (which is about to feature freeze) will be the last version before PHP 8 (except for a deprecations only PHP 7.4). A feature as major as this would almost certainly be rolled into a major version, and there are a lot of significant features already tabled for PHP 8 with no mention of cleaning up the core functions.

If we want to “Make PHP Great Again”, or at least “Make PHP Slightly Better Before 2025”, now is probably a good time to start banging that drum!

A Beginners Guide to Beating the Bookmakers with TensorFlow

Tue, 20 Feb 2018 09:00:00 +0000

Alan - It says here we should use TensorFlow. Apparently we can use a Deep Neural Network to predict the outcome of football (soccer) matches. Who wants to help me collect training data?

Doug - I don't think you should be doing so much gambling tonight Alan.

Alan - Gambling? Who said anything about gambling? It's not gambling if you know you're going to win? Deep Neural Networks are a foolproof system.

Stu - It's also illegal.

Alan - It's not illegal, it's frowned upon. Like mining bitcoin on web browsers.

My pitch for a remake of The Hangover didn’t go down very well, but hopefully the research I did for the script will still be useful to someone.

This article explains how to use the TensorFlow Estimator API to create a simple Deep Neural Network (DNN) that makes predictions about football (soccer) matches. It will assume that you have installed TensorFlow and are familiar with the Python language.

There is a corresponding GitHub repository containing the complete code that will be referenced throughout.

What is a Deep Neural Network (DNN)?

The critical thing for a beginner to understand about a DNN model, is that it is a function. Its purpose is to map a series of input features to an output. When we train our model, we provide a large number of examples where we know all of the features and the desired output, known as the labels. By analysing how bad our model is at predicting the labels, we can use some crazy calculus to modify its neurons and make it less bad at making predictions. This type of machine learning problem, where our training data is labelled, is a supervised learning problem.

Our Test Data

A quick google search for “football data” returns a link to football-data.co.uk - who would have thought?

Looking only at the English Premier League results, I squished about 9 and a half seasons of data into a single CSV file. The file contains loads of useful fields, including goals, shots, shots on target, bookmaker odds and more. Football Data also provide a column key for the data.

The dataset.py script shows how this file is loaded.

The Dataset class converts the CSV data to an array of processed football results containing the following keys. Not all of the results in the dataset can be used, as there needs to have been 10 games of history for each team before the statistics can be calculated.

[{
  'result': 'H', # Could be H, D or A (for home team win, draw or away team win)
  'odds-home': 1.6, # Decimal odds from Bet365 on a home team win
  'odds-draw': 4.8, # Decimal odds from Bet365 on a draw
  'odds-away': 6.0, # Decimal odds from Bet365 on an away team win
  'home-wins': 6, # Number of wins by the home team in the last 10 games
  'home-draws': 2, # Number of draws by the home team in the last 10 games
  'home-losses': 2, # Number of losses by the home team in the last 10 games
  'home-goals': 16, # Number of goals scored by the home team in the last 10 games
  'home-opposition-goals': 12, # Number of goals scored against the home team in the last 10 games
  'home-shots': 68, # Number of shots made by the home team in the last 10 games
  'home-shots-on-target': 53, # Number of shots on target made by the home team in the last 10 games
  'home-opposition-shots': 20, # Number of shots made against the home team in the last 10 games
  'home-opposition-shots-on-target': 14, # Number of shots on target made against the home team in the last 10 games
  'away-wins': 3, # Number of wins by the away team in the last 10 games
  'away-draws': 4, # Number of draws by the away team in the last 10 games
  'away-losses': 3, # Number of losses by the away team in the last 10 games
  'away-goals': 13, # Number of goals scored by the away team in the last 10 games
  'away-opposition-goals': 17, # Number of goals scored against the away team in the last 10 games
  'away-shots': 32, # Number of shots made by the away team in the last 10 games
  'away-shots-on-target': 13, # Number of shots on target made by the away team in the last 10 games
  'away-opposition-shots': 47, # Number of shots made against the away team in the last 10 games
  'away-opposition-shots-on-target': 21, # Number of shots on target made against the away team in the last 10 games
]}

Classification or Regression

The next step is to look at the options available for the type of predictions our model will make. In this regard, there are two primary model types, and these are known as classification or regression models.

A classification model will output a prediction that the input features match a set of classes. An example of this would be using biometric measurements to predict whether a subject was of class male or female. Here, the output from our model would be a two dimensional vector containing the probability that the subject was male and the probability that the subject was female. All classification probabilities in a prediction sum to one.

A regression model outputs a continuous variable (or a vector of continuous variables). An example of this would be using house features (such as location, size and bedrooms) to predict house price.

We are going to use a classification model. Our three available classes will be home win H, draw D and away win A. If the output from our model for a given set of features was [0.23, 0.65, 0.12], that would correspond to a prediction that there was a 23% chance of the home team winning, a 65% chance of a draw and a 12% chance of the away team winning.

This model choice means that we will be using the TensorFlow DNNClassifier class. If we were solving a regression problem, we would use the TensorFlow DNNRegressor class.

There are other classification and regression classes available from the TensorFlow Estimator API. Some of these use linear mathematical models and some use linear models in combinations with DNNs. It is well worth investigating and understanding these, but none of them sound as cool as Deep Neural Network on your LinkedIn portfolio, so we will skip them for now.

Features

Google Developers have an excellent blog post explaining TensorFlow Feature Columns that goes into good detail about the options available.

Our model will only use the numeric_column that is provided, but there are others available (such as a category_column).

The DNNClassifier class we mentioned earlier requires an array of feature columns to be provided. Looking above to the test data section, we have quite a few features available to select.

Removing the result field (that’s going to be the label) and the odds fields, we are left with the following feature columns:

feature_columns = [
  tf.feature_column.numeric_column(key='home-wins'),
  tf.feature_column.numeric_column(key='home-draws'),
  tf.feature_column.numeric_column(key='home-losses'),
  tf.feature_column.numeric_column(key='home-goals'),
  tf.feature_column.numeric_column(key='home-opposition-goals'),
  tf.feature_column.numeric_column(key='home-shots'),
  tf.feature_column.numeric_column(key='home-shots-on-target'),
  tf.feature_column.numeric_column(key='home-opposition-shots'),
  tf.feature_column.numeric_column(key='home-opposition-shots-on-target'),
  tf.feature_column.numeric_column(key='away-wins'),
  tf.feature_column.numeric_column(key='away-draws'),
  tf.feature_column.numeric_column(key='away-losses'),
  tf.feature_column.numeric_column(key='away-goals'),
  tf.feature_column.numeric_column(key='away-opposition-goals'),
  tf.feature_column.numeric_column(key='away-shots'),
  tf.feature_column.numeric_column(key='away-shots-on-target'),
  tf.feature_column.numeric_column(key='away-opposition-shots'),
  tf.feature_column.numeric_column(key='away-opposition-shots-on-target'),
]

Connecting it all Together

model = tf.estimator.DNNClassifier(
  model_dir='model/',
  hidden_units=[10],
  feature_columns=feature_columns,
  n_classes=3,
  label_vocabulary=['H', 'D', 'A'],
  optimizer=tf.train.ProximalAdagradOptimizer(
    learning_rate=0.1,
    l1_regularization_strength=0.001
  ))

This is where we create our model object. The model_dir parameter is used to save training progress so that you don’t have to retrain your model every time you want to make a prediction. The hidden_units parameter describes the shape of the neural network, [10] corresponds to one hidden layer with 10 neurons in it. We define the output that we expect from our model with n_classes and label_vocabulary. Here we state that the three classes our model will analyse are labelled by H, D or A.

The optimizer parameter is optional and the default value is reasonable for many cases. Here the default value is not used because it was necessary to add regularization to prevent overfitting.

We can train our model using the train() method. A model can be trained indefinitely, but with diminishing returns. Different models descend at different rates, so it is important to watch the loss function to see if and when it has converged (and stopped descending).

model.train(input_fn=train_input_fn, steps=1000)

The input_fn parameter is used to provide training data to the model. There are lots of clever ways to feed data into a model, but using numpy arrays is pretty quick and easy:

train_input_fn = tf.estimator.inputs.numpy_input_fn(
  x=train_features,
  y=train_labels,
  batch_size=500,
  num_epochs=None,
  shuffle=True
  )

In the example above, if we had only 3 training examples, train_features and train_labels would have the structure shown below. The keys of train_features must match the keys used when creating the feature columns array.

train_features = {
  'home-goals': np.array([7, 3, 4]),
  'home-opposition-goals': np.array([3, 8, 6]),
  # ... for each feature
}

train_labels = np.array(['H', 'D', 'A'])

The code in predict.py takes the first 95% of results into train_results and the latest 5% into test_results. These are used to evaluate the accuracy of the model and the performance of the betting strategy.

Developing a Betting Strategy

It is important to know that bookmakers do not optimise odds solely to reflect event probabilities. Fortunately for us, they have to consider markets and their exposure to risk too.

Casual gamblers and football fans are generally motivated by opinions, excitement and loyalty. Thus, most are uninterested in betting that the outcome of a match will be a draw. Bookmakers tend to reduce the odds of popular bets to minimise their exposure to risk, and this makes it easier to be profitable when betting on draws rather than wins.

The betting.py script shows an example betting strategy that utilises this logic. If our model believes that the bookmaker has underestimated the chance of a draw by 5% or more, it places a bet.

The Results

The above graph shows how the model developed as it was trained. The DNN converges quickly on a solution and achieves an accuracy of 51%. This accuracy is slightly below the performance of Bookmakers, Bet365 achieves around 53-54% on the same results.

The betting return is less stable, and is in the region of a 50% return on investment. This strategy simulation placed 35 bets whilst considering 172 possible matches. It is difficult to generate an accurate betting return figure, as the strategy places a statistically small number of bets and there is not much test data.

The odds used were only from Bet365, and further gains could probably be achieved by shopping around. That said, Bet365 often provided the most competitive odds available in the dataset.

If you would like to learn more about machine learning, I can highly recommend Stanford University - Machine Learning by Andrew Ng. I am about halfway through the course at the time of writing, and thoroughly appreciating its bottom up approach to these subjects.

You may also be interested in my previous article on object detection and recognition in images with TensorFlow.

Annotating Large Datasets with the TensorFlow Object Detection API

Wed, 24 Jan 2018 22:00:00 +0000

Detections of race cars, after training on a small dataset containing only 60 images

When building datasets for machine learning object detection and recognition models, generating annotations for all of the images in the dataset can be very time consuming. These annotations are required to train and test a model, and they must be accurate. For this reason, human oversight is required for all of the images in a dataset. However, that does not mean that machine learning models cannot be of assistance.

Checking and correcting a set of mostly correct annotations is generally a less time consuming task than creating a complete set of new annotations. When working on a dataset containing thousands of images, saving a few seconds per image could save several hours of work.

The purpose of this article is to prove that for object recognition and detection tasks where high accuracy is not a necessity, small datasets and “out of the box” models can provide useful results.

Using the example of detecting race cars in an image, this article will guide through the following steps.

Annotating images in a small dataset.
Training a primitive model from this dataset.
Use this primitive model to predict the annotations on images from a new dataset.

This article has a repository on GitHub that contains some example code and data. It will be assumed that the TensorFlow Object Detection API has been installed.

Annotating Images

An easy format to use for image annotations is the PASCAL VOC file format. This is an XML file format used by Image Net. The LabelImg program is an excellent tool that can be used to generate and modify annotations of this format.

The data directory in the example repository shows annotations generated using this method. The label map file will need to be created manually, as this is not produced by LabelImg.

Training the Primitive Model

The TensorFlow Object Detection API provides detailed documentation on adapting and using existing models with custom datasets.

The basic process for training a model is:

Convert the PASCAL VOC primitive dataset to a TFRecord file. The example repository provides a python script that can be used to do this.
Create an object detection pipeline. The project hosts official documentation on how to do this and there is an example in the repository. The example in the repository is based upon the ssd_mobilenet_v1_coco checkpoint, there are a number of provided checkpoints that can be downloaded from the official documentation.
Train the primitive model. This can be done locally or on the Google Cloud Platform. The results obtained in this article were from around 10,000 steps.
Export the last checkpoint to an inference graph.

The end result of this process will be a file named frozen_inference_graph.pb. This is the model that can be used to predict annotations.

Predicting Annotations

The pascal-voc-writer library can be used to generate annotations in the PASCAL VOC file format. It has a simple to use API, and the code below shows an example of adding a ‘cat’ annotation to an image.

from pascal_voc_writer import Writer

writer = Writer('path/to/img.jpg', 800, 400)
writer.addObject('cat', 100, 100, 200, 200)
writer.save('path/to/img.xml')

The annotate.py file in the example repository uses this library to adapt the inference example from the official documentation to create PASCAL VOC annotation files rather than image visualisations.

The threshold score for an annotation to be produced can be optimised to suit the dataset and the needs of the operator. The threshold score should balance the frequency of unhelpful annotations against the miss rate. If removing unhelpful annotations is easier for the operator than generating missed ones, a lower threshold score should be used to reflect this.

Below are three more predictions from the primitive model. Despite having a very small dataset and only a few training steps, the model has made useful predictions that would save time annotating these images.

In this example two annotations are correctly suggested and one is missed. The suggested annotations on the furthest car could be shrunk slightly.

These images take a long time to annotate manually. The primitive model has done a reasonable job at cutting out the bulk of the workload. It has struggled in the case of some of the obscured cars, but some of the obscured cars would be difficult for humans to spot.

Both cars are identified, however the bounding box is far too wide for the closest car.

Monitoring Road Traffic with Python

Thu, 24 Aug 2017 22:00:00 +0000

Today, one of my colleagues located a nearby traffic camera that monitored a particularly busy road near where we work. This road can make the journey home a real pain in the ass, and it is much easier to just stay in the office and work later until it calms down a bit.

Here is a sample image from the camera:

There is a lot of content in this image that is not relevant to the amount of traffic in it. A reasonable first task is to extract the important section of road from the background. The Python library Pillow makes this very easy.

The only fiddly bit is finding the coordinates of the polygon that contains the important section of road.

from PIL import Image, ImageDraw

POLYGON_MASK = [
    (0, 0),
    (0, 288),
    (44, 288),
    (140, 125),
    (160, 125),
    (164, 288),
    (352, 288),
    (352, 0),
]

CROP_RECT = (44, 125, 164, 286)


def process(file_path):
    # Open the file path with PIL.Image
    image = Image.open(file_path)

    # Use PIL.ImageDraw to fill in a black polygon
    ImageDraw.Draw(image).polygon(POLYGON_MASK, fill=(0, 0, 0))

    # Crop the image to get ride of unnecessary pixels
    cropped_image = image.crop(CROP_RECT)

    # Let's look at the result!
    cropped_image.save('output.png')

The result:

The next task is to somehow calculate how much traffic there is in this image. Counting cars is beyond the skill level of this author, but counting pixels is the kind of nasty hack that is right up his street (get it?).

OpenCV provides an excellent edge detection algorithm that outputs a monochrome image with white pixels for identified edges and black pixels otherwise.

from PIL import Image, ImageDraw
import cv2
import numpy

# ...


def process(file_path):
    # Same as last time
    image = Image.open(file_path)
    ImageDraw.Draw(image).polygon(POLYGON_MASK, fill=(0, 0, 0), outline=True)
    cropped_image = image.crop(CROP_RECT)

    # Use numpy to convert the image from PIL format to OpenCV
    opencv_image = cv2.cvtColor(numpy.array(cropped_image), cv2.COLOR_RGB2BGR)

    # Apply the Canny edge detection algorithm
    edged_image = cv2.Canny(opencv_image, 100, 200)

    # Let's look at the result!
    cv2.imwrite('output.png', edged_image)

The result:

A useful metric for traffic would simply be the ratio of white pixels to black pixels.

A slightly better metric would be the ratio of white pixels to pixels that could have been white, remember that some of the pixels from the image were black already from the polygon mask.

from PIL import Image, ImageDraw
import cv2
import numpy

POLYGON_MASK = [
    (0, 0),
    (0, 288),
    (44, 288),
    (140, 125),
    (160, 125),
    (164, 288),
    (352, 288),
    (352, 0),
]

CROP_RECT = (44, 125, 164, 286)


# Count the number of pixels that were not masked by the polygon
# (The result of this function will be constant for all images)
def count_unmasked_pixels(pil_image):
    pixels = pil_image.load()

    count = 0

    for x in range(0, pil_image.size[0]):
        for y in range(0, pil_image.size[1]):
            if pixels[x, y] != (0, 0, 0):
                count = count + 1

    return count


# Count the number of white pixels in the edge image
def count_edge_pixels(opencv_image):
    count = 0

    width, height = opencv_image.shape[:2]

    for x in range(0, width):
        for y in range(0, height):
            if opencv_image[x, y] == 255:
                count = count + 1

    return count


# Stitch together and return the ratio
def calculate_traffic_score(file_path):
    image = Image.open(file_path)
    ImageDraw.Draw(image).polygon(POLYGON_MASK, fill=(0, 0, 0), outline=True)
    cropped_image = image.crop(CROP_RECT)

    unmasked_pixels = count_unmasked_pixels(cropped_image)

    opencv_image = cv2.cvtColor(numpy.array(cropped_image), cv2.COLOR_RGB2BGR)
    edged_image = cv2.Canny(opencv_image, 100, 200)

    edge_pixels = count_edge_pixels(edged_image)

    return float(edge_pixels) / float(unmasked_pixels)

Results

Running this algorithm on the camera near my work gives reasonable results for just a few hours of experimentation.

One of the weaknesses of this algorithm is that more edges will be detected in direct sunlight because of the increased contrast. A solution to this would be to use tensorflow or a more intelligent form of processing to actually count the vehicles.

PSR-7 Objects Are Not Immutable

Sun, 22 May 2016 08:00:00 +0000

Wait, what!? But it says they are, right here:

“Why value objects? The proposal models messages and URIs as value objects. Messages are values where the identity is the aggregate of all parts of the message; a change to any aspect of the message is essentially a new message. This is the very definition of a value object. The practice by which changes result in a new instance is termed immutability, and is a feature designed to ensure the integrity of a given value.”

An Experiment

If you were to install the Zend Expressive skeleton (v1.0.0) and register an action that does the following:

$action = function ($rq, $rs, $next = null) {
    $response->getBody()->write('You should not see this message');
    $response = $response->withHeader('Content-Type', 'application/json');

    throw new \Exception('Access denied');

    return $response;
}

You shouldn’t expect to see the message that was written to the response actually rendered. The code we have written explicitly avoided returning the response by throwing the exception. However, if you disable the Whoops error handler and use the default templated error handler you will find that your message still appears at the top of the error page that is generated by the framework.

What’s Happening Here?

What’s happening is that the Zend Expressive framework is rendering the error page to the same object that you wrote your message to. Whilst the actual message object itself is immutable, the body stream that it references is not. Even when this object is cloned or “modified” (to become a new object) it will still use the same stream.

What Does This Mean

What this means is that we lose the core benefit of immutability. That is, we can not trust that the state of our PSR-7 objects cannot be modified when passed to external actors. Alternatively put, we cannot treat these objects as if they were completely immutable value objects.

This topic was actually well discussed whilst PSR-7 was being developed. The meta-document even contains a section explaining the design decision behind why streams in PSR-7 are mutable.

Regardless of the information out there, it appears that there is a lot of code being written that is making this false assumption of immutability.

The Middleware Debate

At the moment, there is an on-going debate about the standardisation of PSR-7 middleware. Two important blog posts have been written about the topic, both of them make the mistake of assuming that PSR-7 objects are completely immutable.

In the post by Anthony Ferrara, he offers the following as an alternative to passing a response instance through to the handle() method:

class MyMiddleware implements Middleware {
    private $response;
    public function __construct(ResponseInterface $response) {
        $this->response = $response;
    }
    public function handle(
        RequestInterface $request,
        Frame $frame
    ): ResponseInterface {
        return $this->response->withStatusCode(404);
    }
}

For this code to work, the response object would need to be a suitable factory for itself. Because of the stream issue, this is not the case and the middleware is not stateless. This is not actually a very serious bug, and it would only become an issue if you tried to call the handle() method more than once (see PHPFastCGI). I should say that the final and preferred solution suggested by Anthony Ferrara does not suffer from this bug and is, in my opinion, the way forward.

The bug is more serious in the post by Woody Gilk. Here he provides an example of some middleware that is designed to handle exceptions:

class ExceptionHandler
{
    public function __invoke(
        ServerRequestInterface $request,
        ResponseInterface $response,
        callable $next
    ) {
        try {
            return $next($request, $response);
        } catch (Exception $e) {
            return $this->withException($response, $e);
        }
    }

    private function withException(ResponseInterface $response, Exception $e)
    {
        $response = $response->withStatus(500);
        // ... additional code to write response body
        return $response;
    }
}

In the description, he even says:

The critical thing to note here is that the partial response is never used. If an exception occurs, the response that was passed to this middleware is decorated and returned. So long as your middleware stack has this middleware as close to the top as possible, a badly formatted response will never get decorated as an error.

As we know from Zend Expressive, this is not the case. Anything written to the response body will actually be appended to anything that was written to the response before the exception was thrown. Again, these are not lone mistakes and there are plenty more (1, 2) examples of the immutability assumption bug.

95% of PHPixie Installs on Packagist are Fraudulent

Mon, 09 May 2016 08:00:00 +0000

Background

To be transparent, I have already had a run-in with Roman Tsjupa (the author of PHPixie, aka dracony) after I exposed his use of sock-puppet accounts on reddit.

The other day he posted this tweet claiming that PHPixie had just received 40 GitHub stars in 24 hours. Knowing his character, I checked and found that all of the accounts were obvious bots that had been registered on that very day and starred only PHPixie projects (proof 1, proof 2). These accounts have since been deleted as I reported them to GitHub.

This got me thinking, how far does PHPixie take the deception and fraud? I had always been suspicious of the number of installs that the project claimed, given that the community is pretty inactive and there is only really one project contributor.

Packagist Fraud

As I am writing this, the project comes second on Packagist searches for ‘framework’ and ‘orm’ (behind Laravel and Doctrine) - with close to 100,000 installs on many of its active repositories.

Suspecting the obvious, I set up a small script to collect some data that I could use to compare the install pattern to those of well known frameworks. The graph below shows the results of this analysis. The downloads figure has been normalised so that the frameworks can be easily compared.

As you can see, every day at around 2AM the number of installs jumps significantly then flat lines. Packagist does implement some protection against this type of fraud. I am not going to go into detail about what this protection is, but let it suffice to say that this install pattern can only be explained by someone actively attempting to bypass this protection. In other words, 2000 fully functional tests on a cron job would not affect the install statistics.

Doing the Math

Packagist has a helpful API for extracting daily installs and the PHPixie data can be found here in JSON format.

As you can see, there have been some days where the scripts have not been run. On these days, there are 2-10 installs rather than 1000-2000. This could just be because a laptop or computer was turned off rather than left on over night. For whatever reason, it gives us an indication of how many actual downloads the PHPixie project should have.

By doing a back of the notepad calculation, using the number 5 to replace the install count on days when the scripts have run, I estimate that over 95% of PHPixie installs have been faked.

PSR-6 Has Serious Problems

Mon, 07 Dec 2015 20:00:00 +0000

UPDATE: Since publishing this article I have been speaking with Larry Garfield, the editor of PSR-6. He has posted his response in the comments section below.

I accept the point that it would be impossible to construct a cache proposal that was approved by all of the voting members that I have mentioned. However, I still believe that the exception definitions are not suitable for ratification and that implementations should not need to supplement the specification with public methods. My personal opinion remains that the proposal is not ready, but I encourage you to read Larry’s response and come to your own conclusion.

PSR-6 is the proposal from the PHP-FIG (PHP Framework Interop Group) of a set of cache interfaces that can assist programmers in writing decoupled code. The idea is that cache drivers can implement these interfaces and then code can depend on an interface and not the underlying cache driver or system.

Here are the proposed interfaces:

<?php

namespace Psr\Cache;

interface CacheItemPoolInterface
{
    public function getItem($key);
    public function getItems(array $keys = array());
    public function hasItem($key);
    public function clear();
    public function deleteItem($key);
    public function deleteItems(array $keys);
    public function save(CacheItemInterface $item);
    public function saveDeferred(CacheItemInterface $item);
    public function commit();
}

interface CacheItemInterface
{
    public function getKey();
    public function get();
    public function isHit();
    public function set($value);
    public function expiresAt($expiration);
    public function expiresAfter($time);
}

interface CacheException {}

interface InvalidArgumentException extends CacheException {}

Problem 1

Some of most important projects in the FIG (regarding caching) have voted against it

One of the first ‘-1’ votes on the proposal at the moment is a resounding no from the Doctrine team. The comment on the voting thread is:

“Unanimous -1 from Doctrine among all core developers. List of deficiencies too big and not pertinent to this thread.”

The Doctrine team are the maintainers of (according to packagist) the most popular caching library in the PHP ecosystem.

Guzzle and Laravel also maintain popular caching libraries and voted ‘-1’. Other member projects in the FIG group should have alarm bells ringing loudly in their ears when they see these votes.

The proposal should be moved back and the concerns of Doctrine, Guzzle and Laravel should be listened to.

Problem 2

(See Mailing List)

The second problem is with the last two interfaces. They fail to make it clear that they are interfaces and they also pretend to be exception classes. This might provide a minor irritation to anyone implementing the interface, but to the user it should not be a problem. You can still catch it the same, right?

The exception system in PHP was designed to be extended. The issue here is that there is no guarantee that a class implementing the InvalidArgumentException interface defined by the proposal will also extend the root InvalidArgumentException class defined by PHP.

This leads to the following ridiculous situation which is not only plausible, but completely in agreement with the specification:

if (
     $exception instanceof Psr\Cache\InvalidArgumentException &&
    !$exception instanceof \InvalidArgumentException
) {
    echo 'Huh?';
}

If you wanted to guarantee catching all invalid argument exceptions in a block of code you would need to do this:

try {
    // code
} catch (Psr\Cache\InvalidArgumentException $exception) {
    // invalid argument exception logic
} catch (\InvalidArgumentException $exception) {
    // invalid argument exception logic again
} // ...

This might not sound so significant, but I cannot for the life of me understand why the proposal decided to use interfaces in the first place. It doesn’t appear to solve any problems and it actually creates one.

Problem 3

(See Mailing List)

The cache item interface lets you set an expiration time using the following methods;

$item->expiresAfter(300);
$item->expiresAt(new \DateTime(...));

However, there is no method available to retrieve this expiry information from the item. There are good reasons for this. Primarily, if you retrieve an item from memcached it is not actually possible to find out when it expires, at least not without some serious wizardry and expensive operations. This is the case for many other cache systems too.

On the surface this is not a problem. If you cannot know the expiration of an item, there shouldn’t be a method promising to provide it.

The problem only becomes apparent when you think of how the cache pool handles this information; after all, it is the cache pool that actually persists the cache item. Somehow, the cache pool needs to know the expiration time that you set on the cache item.

Now there are two options here for an implementation.

The first is that the cache item could be provided a method of communication with the cache pool on construction. However, this leads to a new problem, memory leaks.

When the item falls out of scope the cache pool will have no way of knowing that it needs to forget all the information it was storing about the cache item. The only way to control the memory leak in a long running process would be to continuously recreate the cache pool.

The second option is that the implementation could expose a public method on the cache item that is not defined on the CacheItemInterface. This is a particularly unsatisfying solution for a few reasons:

Public methods are a pain. As soon a class in a library has a public method it cannot change that without breaking backwards compatability. The last thing any library developer wants to do is create unnecessary public methods that they then have to support until the next major version.
Users should not be calling such a method. If implementations do add a public getExpiration() method, this will ultimately not be the method that users expect it to be (for precisely the reasons that it was removed from the specification in the first place). When you create a method that users should not be using… it should not be public.
The specification should be complete for both calling code and implementations.

One solution to this specific problem would be to set the expiration time when saving the object. Something like this:

$pool->save($item, 300);

Summary

Anthony Ferrara’s Open Letter To PHP-FIG describes well the core issue of over engineering in PSR-6.

A lot of work has gone into PSR-6 and I can understand everything the authors have tried to do. That said, I think it would be a huge mistake for the PHP-FIG if the proposal was to pass in its current form.