## Learning more about neural networks

Cheap and accessible books on neural nets are not easy to find – so I bought “Practical Neural Network Recipes in C++” as a second-hand book on Amazon (link). According to Google Scholar this book – though now 24 years old – has over 2,000 citations, so it ought to be good, right?

Well, the C++ is really just C with a few simple classes – lots of pointers and not an STL class to be seen (but then I can hardly blame author Timothy Masters for not seeing into the future). The formatting is awful – it seems nobody even thought that you could put the source into a different font from the words. But, yes, it works.

Essentially I used it – though I had to get additional help to understand back propagation as the book’s explanation is garbled, mixing up summed outputs and outputs from activation functions for instance – to build a simple neural network which worked, after a fashion.

(In fact I didn’t build a fully connected network, because the book didn’t say – anywhere that I could see, anyway – that you should. I have rectified that now and my network is much slower at learning but does seem, generally, to be delivering better results.)

But it seems that 24 years is a long time in the world of neural nets. I now know that “deep learning” isn’t just (or only) a faddish way of referring to neural networks, but a reflection of the idea that deep nets (i.e., with multiple hidden layers) are generally thought to be the best option, certainly for image classification tasks. Timothy Masters’s book essentially describes additional layers as a waste of computing resources: certainly anything above two hidden layers is expressly dismissed.

Luckily I have access to an electronic library and so haven’t had to buy a book like “Guide to Convolutional Neural Networks” (Amazon link) – but I have found it invaluable in learning what I need to do. But it’s complicated: if I build many different convolutional layers into my code the network will be slow(er) – and it will be time to break out the threads and go parallel. But now I have fallen into this rabbit hole, I might as well go further.

## First results from the “musical” neural network

I am working on a project to see whether, using various “deep learning” methods, it is possible to take a photograph of some musical notation and play it back. (I was inspired to do this by having a copy of 1955’s Labour Party Songbook and wondering what many of the songs sounded like.)

The first task is to identify which parts of the page contain musical notation and I have been working with a training set built from pictures of music chopped into 100 x 100 pixel blocks – each is labelled as containing or not containing musical notation and the network is trained, using back propagation, to attempt to recognise these segments automatically.

Now I have tested it for the first time and the results are interesting – but a bit disappointing. In this image all that is plotted is the neural net’s output: the redder the image, the higher the output from the net’s single output neuron:

It’s a bit of a mystery to me as to why you can see the staves and notes in this sort of shadowy form: as that means the network is rejecting them as musical notation even as it does highlight the regions where they are found as the best places to look.

To make it all a bit clearer, here are the results with the blue/green pixels of the original image unchanged and the red pixels set on the strength of the network’s output:

It seems clear the network is, more or less, detecting where there is writing on the page – though with some bias towards musical staves.

I’m not too disappointed. My approach – based on stuff I read in a book almost 25 years old – was probably a bit naïve in any case. I came across a much more recent and what looks to be much more relevant text yesterday and that’s what I will be reading in the next few days.

(You can see the code behind all of this at my Github: https://github.com/mcmenaminadrian)

## Back to neural networks

Neural networks have fascinated me for a long time, though I’ve never had much luck applying them.

Back in the early and mid 1990s the UK’s trade and industry department ran a public promotional programme to industry about NNs and I signed up, I even bought a book about how to build them in C++ (and I read it, though I have to confess my understanding was partial).

My dream then was to apply the idea to politics: as a more effective way of concentrating resources on key voters. My insight was that, when out doing what political parties then called “canvassing” and now – largely for less than fully honest legal reasons as far as I can see – call “voter ID” (in electoral law “canvassing” i.e., seeking to persuade someone to vote one way is more highly regulated than simply “identifying” how they intend to vote) you could quite often tell how someone would respond to you even before they opened the door. There was some quality that told you, you were about to knock on a Labour voter’s door.

I still don’t know what that quality is – and after the recent election I’m not sure the insight applies any more any way – but the point was that if you could take a mass of data and have the NN find the function for you, then you could improve your chances of getting to your potential support and so on…

But I never wrote any code and NNs seemed to go out of fashion.

Now, renamed “machine learning”, they are back in fashion in a big way and my interest has been revived. But I am not trying to write any code that will work for politics.

Instead I am exploring whether I can write any code that will look at a music score and play it back. (This probably a silly idea as a start NN project, as music is not easy to decipher in any way as far as I can see).

I have read another book on C++ and neural networks (and even understood it): an ancient tome from the previous time NNs were in fashion.

The first task is to actually identify which bits of scribbling on the page are musical notation at all. And I have written some code to build a training set for that – here. It might well be of use to you if you need to mark bits of a JPEG as “good” or “bad” for some other purpose, so please do get in touch if you need some help with that.

## Game of Life in Scratch

A few days ago I asked for volunteers to read a book I was writing on programming, using Scratch, MIT’s visual, event-driven, programming environment.

I have not yet had any volunteers, though the flurry of online interest did get me to complete the first draft – so alpha testers still needed.

In the meantime, I have also published the version of Conway’s Game of Life the text is based around. It’s a slightly unusual implementation in that the surface of the game is, in effect, spherical in effect a torus – i.e. the edges are effectively a feature of the projection of the surface but left joins to right, top to bottom and so on.

Update: Been pointed out to me that this isn’t a sphere but a torus, and so I have updated the copy.

## Do you want to learn about coding? Free book offer!

Three years ago I had great fun teaching some primary school children how to use Scratch and take their first steps towards being computer programmers of the future.

At the time there was a lot of optimism about changes to the English curriculum and a return to a more rigorous teaching of computing in schools.

Today, though, there is a pretty predictable warning that things aren’t going to plan – like so much else in the last seven years good intentions were not backed up by the money needed to make the policy really work.

Back then I decided I was going to write a “book” on how to code – my principal audience being teachers who suddenly found themselves with the requirement to teaching basic coding … but had a change of job, couldn’t continue the teaching and nothing much happened.

But, having had another, err, change of job (to currently no job – have a look here if you want to hire me), I’m now determined to finish it and so I am also looking for anyone willing to read it through.

I don’t intend to charge for it, so it’s not a commercial operation, but I would be interested in hearing from anybody who has an interest in learning to code/learning more about computing but who is not an experienced programmer – crudely put, if you think the “book” stinks then I probably won’t make much more of an effort with it, if you think it is good or getting there then I’ll keep working on it, trying to make it e-reader compatible and so on.

## Crazy ideas you have in the bath

You know how it is … you go for a run and then lying in the bath you read a New Scientist article about Dark Energy and you think of two crazy ideas which you hope some respectable scientist will at least have stuck a paper on arXiv on, so you can at least say “I thought of that in the bath and it might even be right…”

Except that you can find no such papers, so you are reduced to looking like the crackpot you are by posing them here:

• Inertial mass is caused by the gravitational field of a certain amount of matter that has been trapped in collapsed dimensions. Those dimensions are always the same distance away from any given point, so inertial mass is the same anywhere in the universe.
• Dark energy is caused by the ‘evaporation’ via Hawking radiation or similar of our universe (sadly I am not the first to have thought of this particular piece of crack-pottery, so I won’t be collecting a Nobel prize for it). Further searching reveals there is even an arXiv paper on such an idea after all.

## Failing software – again

The line above is from a real (and current at time-of-posting) job advertisement for a software developer. I’m not positing it because I think it is bad, shocking or dangerous, but mainly because it is illustrative of the real world: developers are expected to be “pragmatic” when it comes to testing the software they make for correctness.

So when, as happened today with British Airways, we see a major software failure (or at least what looks like a major software failure), maybe we should not be too surprised.

There is more to it, of course, than the fact developers are expected to compromise on testing their code: there is the simple fact that most functions are not computable at all and that for many others we don’t know how to compute them.

For an example of something uncomputable there is Hilbert’s tenth problem:

For any given a Diophantine equation is there a general algorithm that tells us whether or not there is a solution with the unknowns taking integer values…

• A Diophantine equation (named after Diophantus of Alexandria) is a polynomial (e.g., $x = y$, $y^2 + 5x + z^{67} = 0$, $3x + 2y +5x^2y^3 + 90 = 7z$ and so on) equation where all the coefficients of the unknowns are integer (counting number) values and where there are a finite number of unknowns.
• An algorithm is a set of rules of simple mathematical procedures to solve a problem

So the task is to take a Diophantine equation and not to find a solution but to determine whether such a solution exists – but no such algorithm exists and therefore we cannot write any sort of computer program that would do this for us, no matter how powerful a computer we were using.

And for an example of something for which a solution does exist but which we will struggle to find, there is the famoustravelling salesman problem” – namely given a set of cities all connected by roads (or railways, or air routes) what is the most efficient way of visiting all the cities.

This problem is an example of what is known as an “NP” problem – in other words one for which (we believe) no general algorithm exists to produce a solution in “polynomial time” (meaning in a time related to the length of the input – in this case the number of cities).

Plainly a solution does exist – there is a quickest way to travel to all the cities – but the only immediately available way to be certain of finding this is to try all the solutions.

But that will take time: for even moderately sized problems quite possibly more time than we have before the Sun swallows the Earth. To get round this we have to use heuristics (essentially educated guesswork) or approximations.

In the travelling salesman case those approximations are powerful and efficient – that’s why modern logistics works – but the general point remains the same: in many cases we are building software that approximates the solution to our problem but may not answer it fully correctly and, unless there is a breakthrough which eliminates NP problems as a class, we are always going to be stuck with that.

## Switching to Opera

When I first started using it, Google’s Chrome browser seemed like a huge leap forward: a thread for each open tab, what’s not to like about that?

Latterly, though it has felt more and more like a drag: run it for any length of time and your computer will thrash or even freeze as it is such a poor manager of resources and/or a major leaker of memory.

The answer, so I have read, is to use Opera – so that is what I am doing. Will report on progress in a few weeks.

## Let’s talk about sex (dolls)

I’m not all that interested in sex dolls, actually. But what I am interested in is the reactions they provoke from people when they consider the nature of intelligence.

My view – pretty much that followed by Alan Turing in his pioneering paper “Computing Machinery and Intelligence” – from which we get the “imitation game” aka the Turing Test – is that intelligence is whatever looks like intelligence.

The relevance of this to sex dolls is that the BBC’s technology correspondent Jane Wakefield has put together a series of reports on the subject – the first was on “From Our Own Correspondent” last Saturday, there’s a web piece – here – and there is a report for the BBC’s World Service yet to come.

Wakefield argues that while the sex doll “Harmony” can say things which sound like intimate small talk, the doll can never know the feelings behind the words.

But what does that mean? At the most basic level none of us can live inside the head of another – we can never “feel” what it’s like to that other person, because we cannot be them.

Or, to paraphrase Turing, you might like strawberry ice cream and I might hate it: but we are both tasting the same thing, so what does this feeling of “love” or “hate” correspond to? How could you know how I “feel” about the ice cream, when you “feel” differently?

It’s an entirely subjective thing, so how can you assert that the machine “feels” nothing?

Of course, the human brain and human experience generally appears to be a massively parallel thing and we simply cannot, yet, replicate that in a machine, but if we could are we seriously suggesting that human consciousness transcends the material? That simply doesn’t make any sense to me.

## Proprietary software as a false economy

I recently had to fill in a form for the Computer Science Department at the University of York.

Like, I am sure, any computer science department in any major world university, York is a “Unix shop”: research servers all run Linux and I guess the academics who aren’t using that are – as I am now – are running the modified/derived BSD that is Mac OS X.

But the form was “optimised” (i.e., only able to operate properly on) Microsoft Word – not a piece of software found on many ‘nix machines.

Because the rest of the University – like almost all of Britain’s public sector – was totally reliant on Microsoft’s proprietary offerings.

Thirty years ago I worked in a public sector organisation that used a mixture of proprietary software for “mission critical” work – Netware, Word Perfect and MS Dos. But even that mixture has gone: it’s Microsoft for everything (on the desktop) these days.

And now the price of that false economy – because so often this reliance on Microsoft has been justified because it keeps training costs low (“everybody knows how to use it”) – has been revealed by a massive global ransomware attack.

If free/open source software (FOSS) had been more-widely used then, of course, the risk would not have disappeared: not least because the crackers would have turned their attention to FOSS and left Windows behind: but there are two pretty obvious advantages to FOSS in terms of security:

• You can see how it works – you wouldn’t walk across a bridge with no visible means of support, yet every time you use proprietary closed-source software you do just that: the fact it hasn’t fallen down yet seems like a poor justification.
• Everybody can fix it: if Microsoft’s software breaks or is seen to have a vulnerability you are essentially reliant on them to fix it. And if you are using an unsupported piece of software you may not even have that. Again there are no guarantees of invulnerability with FOSS – software is hard – but there is a guarantee that you or anyone you ask/pay can attempt to fix your problem.

It’s time we ended this dependency on proprietary software and invested in a FOSS future.