Sunday, 31 July 2011
Friday, 21 August 2009
Psion pioneered the Symbian operating system that was used on their Series 5 devices but was also popular on many high end smartphones from Nokia and Sony-Ericsson. One of my favourite devices was the Psion Revo as this was the perfect combination of small size but with a usuable keyboard and screen. Being launched in 1999, it was groundbreaking for the time but with only 8MB of storage, a 30MHz ARM processor and no connectivity, it is of limited use in today's media rich and connected world.
After the Psion my next device was a Sony-Ericsson P910i, this is a smartphone also running Symbian but this time with GPRS connectivity, bluetooth, 96MB internal memory and expandable upto 1GB using memory stick pro duo. This was a great piece of kit that offered media, a web browser (Opera) and the usual PDA type functions with great handwriting recognition. I also used this with a separate bluetooth GPS receiver and TomTom Mobile software as a SatNav for the car. The P910 did pretty much everything the iPhone does but 4 years earlier, it's clunky by comparison and is showing it's age now.
Alas, times move on and the P910i suffers from a dated browser that doesn't support Ajax, Flash and many other modern features and connectivity is limited to GPRS which is slow and expensive. So it was time to upgrade but to what ?
The obvious choice is the iPhone but at around £600 (£100 purchase + £30/month) this is a serious outlay on a toy. Also, the browser doesn't support Flash or Java and the development tools are all closed. Furthermore you can't develop on it unless you have a Mac (another £1000 outlay for a computer worth £400 !) and you can't distribute your software unless Apple approves it. There's the Ipod Touch as a cheaper alternative but this has no GPS and still suffers from the closed development environment. So this option is ruled out for me.
Next consideration are the Google Android phones like the HTC Hero which is a much better proposition. Under the covers they run a cut down Linux OS but with a Google Java based application environment on top. The browser supports Flash and Java. Development tools are freely available and there is no restriction on distributing software which is written in Java. Sounds a good option but, it's still expensive at £30/month (£540 for minimum 18 month contract).
So what did I go for, neither. It turns out that Expansys were flogging off the Nokia N810 cheap to make way for the new model which is due out imminently. The N810 is marketted is an 'Internet Tablet' by Nokia but it is closer in function to the original Psion palmtops than to modern smartphones but was perfect for my needs with :-
- Linux OS,
- 128MB RAM + 2GB internal storage + expandable upto 32GB through MiniSD,
- Mozilla Browser with Ajax and Flash 9 support,
- Slide-out keyboard or handwriting recognition,
- Built in GPS and SatNav software with UK maps,
All for £128 (30% cheaper than an Ipod Touch). It has the disadvantage that there is no GPRS or 3G support so the internet can only be used where there is Wifi available but it has what is probably the best browser on any mobile device which can be used to access virtually any site that a desktop browser can. Even the IG Index site which is highly Ajax oriented loads up and works fine albeit more slowly than on a desktop. The screen resolution is a respectable 800 x 480 so this is a genuinely usuable format packed into a small size. The Flash player works fine with YouTube and Magnatune although I've yet to get BBC iPlayer working.
The OS is a fairly standard Linux distribution but with a custom Windowing environment designed for touchscreen and even includes a terminal. There are plenty of applications available through the repository but porting any other open source Unix apps should be fairly trivial so long as the resource requirements aren't too great. There is also a development underway to replace the underlying Linux distro with Ubuntu so you get the benefits of Ubuntu with the touchscreen friendly window manager - perfect.
Physically the device is about the same size as an IPhone but slighter thicker at 14mm and noticeably heavier. It has a sturdy metal case that slides down to reveal the keyboard. It also has an integral kickstand wich mean that the device can stand up which is useful for watching videos, photo slideshows or Skype video calls. It's a well designed, solid, compact design but I still think that a traditional clamshell design like the Psion's would have been better or better still dispense with the physical keyboard and rely on the onscreen keyboard or handwriting recognition. Having said that, my attempts to use the handwriting recognition have so far been disappointing finding it very slow and practically unusable. This is a surprise considering how well it worked on the P910 which had a tenth of the processing power of this device. Perhaps it needs more practice on my part and maybe some tuning so I'll bear with it for the time being.
As an update, I'm loving this device and find that I'm using it more often than my laptop for casual browsing and certainly for media like watching vides on YouTube or listening to music on Magnatune. I've persevered with the handwriting recognition and get a high accuracy rate but it still remains too slow for entering anything more than a couple of words. Sometimes it won't respond at all, other times it lets you get a couple of letters ahead but then just forgets earlier letters so you have to rub out and start again. This is a bitter disappointment to me as the P910 was great, my ancient Compaq Ipaq with PocketPC was acceptable and even the humble Palm Pilot of a decade ago was better than this. So come on Nokia give us an update to this rubbish.
What I really like about this device is that it is a regular Linux machine with everything where you'd expect it. This means that porting software is trivially easy.
Friday, 14 August 2009
Unfortunately MetaTrader is a purely Windows application but the great news is that it installs and runs fine under Wine on Ubuntu 9.04. It will install under vanilla Wine from the Ubuntu repository but unfortunately it won't run because it requires MFC libraries that aren't included in the default Wine installation.
This is easily remedied by copying the following files from a Windows XP box :-
-rwxrwxrwx 1 andrew andrew 924432 2003-07-16 17:28 mfc40.dll
-rwxrwxrwx 1 andrew andrew 927504 2008-04-14 01:11 mfc40u.dll
-rwxrwxrwx 1 andrew andrew 1028096 2008-04-14 01:11 mfc42.dll
-rwxrwxrwx 1 andrew andrew 981760 2007-04-03 04:14 mfc42u.dll
-rwxrwxrwx 1 andrew andrew 1060864 2003-03-18 20:20 mfc71.dll
-rwxrwxrwx 1 andrew andrew 1047552 2003-03-18 20:12 mfc71u.dll
-rwxrwxrwx 1 andrew andrew 22528 2008-04-14 01:11 mfcsubs.dll
to your .wine/dosdevices/c:/windows/system32 directory.
Wednesday, 15 July 2009
It sounds rather obvious but you should be successful if you win more on your winning bets than you lose on your losing bets even though you might lose more often than you win. So the emphasis should be on profit rather than the number of times you win or lose. Reading books like 'The Naked Trader', it seems the biggest danger is your emotions forcing you to throw good money after bad when you are losing and making you cash in your winning bets too soon when they're in profit rather than letting them grow to reach their full potential.
Using software should remove the emotions providing the user follows the orders exactly and is not tempted to intervene. So I set about writing some software, in Scheme obviously, to generate a single market order for a bet on the FTSE100 index the next day based on the previous day's data. Here's an example of the daily movement :-
The FTSE is an ideal target for spread betting because it has a large daily movement range and the spreads offered by most companies are very tight (e.g. IG Index offers only 2 points) and you can have a stop that is only 30 points away which means that can very cheaply place bets.
The order might say Buy at 4280 with a stop of 30 points, no limit and a rate of £1 per point. If the market is lower than 4280 at any point during the day, your bet will be placed. For every point that the market moves up, you will gain a pound. If the market then moves downwards you will lose £1 for every point until it reaches the 30 point stop in which case you've lost your bet and are £30 poorer. If on the other hand the market had climbed 100 points you'd be almost £100 richer (the difference is the spreads taken by the spread betting firm). If the market never came down to 4280, then your bet would not be placed and you wouldn't gain or lose anything.
So the first task was to build a simulator and get hold of some historic 'intraday' data i.e. minute by minute movements in the price going back over three months. The simulator takes a list of orders and runs them in a particular day to calculate the profit or loss for each order. Wrapped around this is a harness that for each day in the set will call out to a strategy function asking for orders based on the previous days statistics. The daily and weekly profitability is then calculated for the strategy so you can decide whether it's a winner or not. You can then use the strategy to generate orders for the next day i.e. tomorrow, hopefully with confidence that it will win more than it will lose.
Using the simulator, you can try out strategies that you think might work - here's one that's not bad :-
;; Stats used by strategy function
;; A very simple but surprisingly effective strategy
(define (simple-strategy stats)
(let ((gain (- (stats-prev-close stats) (stats-prev-open stats)))
(range (- (stats-prev-high stats) (stats-prev-low stats)))
(mid (/ (+ (stats-prev-high stats) (stats-prev-low stats)) 2)))
((> gain 10) (list (make-order - (+ mid 5) 30 0 1.0)))
(else (list (make-order + (+ mid 10) 30 0 1.0))))))
The simulator shows that this very simple strategy generates around 500 points profit over the sample three month period. (NB. past performance is no gaurantee of future performance !). Make-order takes the direction (+ for buy order or - for sell), the opening (order limit) value, stop distance, the limit distance and price per point. Note that the only input data the strategy uses is the previous day's opening, closing, high and low price. The strategy works on the basis that if the FTSE gains one day then it'll probably come down the next. Obviously this isn't a universal rule and will very often be wrong but when you're wrong you will lose at most 30 points and when you're right you could gain in excess of 100 points.
Using manual analysis of the charts and computing some optimum values, I created a variation on the one above that managed to average around 1000 points profit for the period but I had no way of knowing if it was optimal or not.
Genetic programming is a technique for evolving programs to solve a problem. Here our problem is to come up with a trading strategy by creating program fragments that breed and mutate over time and are evaluated against a 'fitness function' the least fit are discarded and the most fit are encouraged to breed to produce stronger offspring. In our case the fitness function is the simulator - the total profit over the three months data is all that matters. If you make a good profit you can stay and breed, if you don't you're out. Breeding (crossover) takes complete elements at random from both parents. Mutation randomly and subtly modifies one of these elements to produce something different. After many generations you get a very optimal solution to the problem you are trying to solve.
Scheme is the perfect language for genetic programming because :-
- You can modify code easily by manipulating lists,
- You can evaluate your new code with eval.
Which basically is a collection of expressions that are evaluated to produce the corresponding values required for the order. The profit member is used to track which of the traders are successful.
There are a population of traders that exist in a list that are initialised with random expressions :-
(define (pick-from lst)
(list-ref lst (random-integer (length lst))))
(pick-from '(o c h l g r m)))
(define (get-random-constant x)
(let ((r (- x (random-integer (* x 2)))))
(if (= r 0) (get-random-constant x) r))) ;; Don't want any divide by zero errors
(make-trader (pick-from '(+ -)) (get-mutated-term (get-random-var)) (get-mutated-term (get-random-constant 100)) (get-mutated-term (get-random-constant 100)) 0))
(define (initialise-traders num)
(let ((traders '()))
(let nxt ((x 0))
(if (< x num)
(set! traders (cons (make-random-trader) traders))
(nxt (+ x 1))
Breeding and Mutation
The following function breeds two traders by selecting features from each and then adding mutation :-
(define (breed t1 t2)
;; Create child from parents
(let ((t (make-trader
(pick-from (list (trader-dir-func t1) (trader-dir-func t2)))
(pick-from (list (trader-start-func t1) (trader-start-func t2)))
(pick-from (list (trader-stop-func t1) (trader-stop-func t2)))
(pick-from (list (trader-limit-func t1) (trader-limit-func t2)))
;; Add mutation
(let ((r (random-integer 5))) ;; what to modify
((= r 1) (trader-dir-func-set! t (mutate (trader-dir-func t))))
((= r 2) (trader-start-func-set! t (mutate (trader-start-func t))))
((= r 3) (trader-stop-func-set! t (mutate (trader-stop-func t))))
((= r 4) (trader-limit-func-set! t (mutate (trader-limit-func t))))))
You can see that one (or none) of the trader functions is selected for mutation which is done with these functions :-
(define (mutate-list l)
(map (lambda (x)
(if (> (random-integer 10) 2) (mutate x) x))
(define (get-mutated-term t)
`(+ ,t ,(get-random-constant 100))
`(- ,t ,(get-random-var))
`(* ,t ,(get-random-constant 100))
`(/ ,t ,(get-random-var))
`(if (> ,t ,(get-random-constant 100)) ,t ,(get-random-var)))))
(define (mutate f)
((list? f) (mutate-list f))
((number? f) (pick-from (list (get-mutated-term f) (+ f (get-random-constant 10)))))
((or (eq? f 'o) (eq? f 'c) (eq? f 'h) (eq? f 'l) (eq? f 'g) (eq? f 'r) (eq? f 'm)) (get-mutated-term f)) ;; Variable
((or (eq? f '+) (eq? f '-)) (pick-from (list '- '+ `(if (> ,(get-random-var) ,(get-random-constant 100)) + -))))
((or (eq? f '*) (eq? f '/)) (pick-from '(+ - / *))) ;; Arithmetic operator
((or (eq? f '>) (eq? f '<) (eq? f '=)) (pick-from '(> < =))) ;; Logical operator
(else f))) ;; No mutation - don't know what to do
The key function here is mutate which takes an expression term and mutates it using get-mutated-term.
The rest of the code is concerned with maintaining the population pool, evaluating the profitability of the members and then discarding the least profitable and breeding the most profitable.
The simple static example above achieved a total profit of 509 points over the 3 months data, the genetically bred trader achieved 1469 ! My efforts to hand optimise the simple trader resulted in a profitability of around 1100 but the genetic model has even exceeded this by 30%.
Update: I expanded the data available to the traders to include a long and short moving average and also the gain on the Dow Jones Industrial Average and re-ran the process. The winning trader had a profitability of around 1700 points but surprisingly hadn't chosen to use the Dow variable.
I've now created a separate blog to document my efforts to take this study forward to fully automated forex trading, please visit http://www.thegenetictrader.com to check on my progress.
Friday, 22 May 2009
Using Apache as a front end web server isn't strictly necessary since the Black Hole web server fully supports HTTP and is capable of serving static as well as dynamic content but for robustness, SSL support and to offload the serving of static content it makes sense to place Gambit behind Apache.
Gambit uses lightweight internal threads which allows it to efficiently support thousands of concurrent requests but these run in a single OS thread. This means that if you are running on a multicore or multiprocessor system, you will need to run multiple Gambit processes and load balance between them in order to fully utilise the available CPU. The load balancer would need to be 'sticky' unless you are going to serialize continuations outside of Gambit.
Enter Black Hole
Black Hole is primarily a R5RS compatible module system for Gambit which allows you to easily import libraries into your code including macros which has previously been tricky on Gambit. Conversely you can also export your own code and create your own libraries to fully modularise your development. This is basic stuff but the lack of it has really stifled development on Gambit. For the web developer, the most interesting aspects are the bundled libraries for web serving, SXML and the 'Spork' continuations based web framework.
This guide is based very much on Per Eckerdal's Black Hole Tutorial guide but has been extended based on my own experiences using Black Hole.
Installing Black Hole
Assuming that you have Gambit installed, you need to download the latest Black Hole code from the git repository :-
git clone http://mwaza.dyndns.org/apps/files/modules.git
That will clone the current code repository to 'modules' directory in your working directory, you now need to copy or symlink this directory to the Gambit lib directory e.g. C:\Gambit-C\v4.4.3\lib\ on Windows to create ~~/lib/modules.
Although not strictly necessary, now would be a good time to build all the modules :-
Black Hole is now installed and ready to use, simply do a (load "~~/lib/modules/build") in Gambit to initialise. It is recommended that you use gsc rather than gsi because Black Hole uses compilation features that aren't available in gsi.
For convenience, you can create a wrapper command that loads Black Hole at startup, e.g. on Windows :-
gsc -:dar- -e "(load \"~~/lib/modules/build\")" %1 -
Copy this to your Gambit bin directory then you can use 'bsc' in future instead of gsc or gsi.
Setting Up Apache
In this scenario we are using Apache to proxy the requests through to Gambit and to serve any static content, here's a sample Virtual Host setup from httpd.conf :
ProxyPass /spork http://localhost:8080
ProxyPassReverse /spork http://localhost:8080
This configuration will forward any requests with a URI that starts with '/spork' to the Gambit server running on port 8080. Any other requests will be served from the 'www' directory under Apache.
Creating your Spork Application
So you are now ready to go, fire up Eclipse (with SchemeScript) or your alternative favorite development tool. Start your Gambit REPL window (with bsc or load the modules manually).
Enter this into REPL :-
(import (std spork/core))
(define c (spork-serve root: "/spork" ))
(add-spork c ("one")
(title "Hello, world!"))
"This is my first web application using Spork :)")))
And fire up your browser :-
OK it's not a spectacular example but you are setup now ready to do some more serious development effortlessly through REPL - just redefine the function and retest with no need for a restart.
Wednesday, 29 April 2009
After a few weeks wrestling with the MySQL protocol, I've finally managed to produce a reasonable attempt at a client for Gambit (and probably other Scheme's).
This is a pure Scheme implementation rather than using Gambit's C interface to glue the existing MySQL C API into Gambit. The reason for this is that Gambit's power, and in my view fantastic potential as a web platform, comes from it's lightweight threads which allows Gambit to process thousands of concurrent requests in a single OS thread. This is all acheived by an internal schedular, continuations and very careful use of non-blocking calls on all I/O functions. Using the C API effectively takes the OS thread out of Gambit and into the C code and therefore all of those lovely threads stop executing until your call is completed - seriously bad news for a web application.
This library is now around 80% complete and supports the following features :-
- Multiple concurrent connections to MySQL,
- Authentication using both the old and new authentication methods,
- Switching Schema's,
- Dynamic queries, updates, deletes etc.
- Prepared statement creation,
- Executing prepared statements (queries and updates) with bind variables for most datatypes,
- Closing prepared statements,
- Closing connections.
There are a few limitations with the driver, i.e. features that aren't implemented yet :-
- Not all datatypes are supported yet in prepared statements e.g. floats, doubles, sets, enums and one or two others. Some of these can easily be avoided e.g. by using DECIMAL instead of FLOAT or DOUBLE.
- BLOB's and large character fields are not yet supported in either dynamic or prepared statements.
- Large result sets requiring supplementary fetches aren't supported as I haven't implemented the additional fetch function.
The library is in the Gambit Dumping Grounds. Here's an example of how you use it :-
(with-connection (sql-connect "localhost" 3306 "test" "fred" "fred")
;;(mysql-execute "drop table test")
(mysql-execute "create table test (id integer PRIMARY KEY AUTO_INCREMENT, name varchar(50), inserted datetime, amount decimal(10,2) DEFAULT -4.5)")
(pp (sql-execute "select * from test"))
;; Prepared Statements
(let ((stmt1 (sql-prepare-statement "insert into test (name, amount, inserted) values (?, ?, sysdate())"))
(stmt2 (sql-prepare-statement "select id, name, amount, inserted from test where amount > ? order by id")))
(pp (sql-execute-prepared-statement stmt1 '("Fred" 10.5)))
(pp (sql-execute-prepared-statement stmt1 '("Joe" 5.5)))
(pp (sql-execute-prepared-statement stmt2 '(4.5)))
;; Close connection
Result sets are returned as a list of lists i.e. a list of column values for each row in a list for the result set. The first row in the list is the column headings. For dynamic statements, the column data is always a string regardless of the SQL type because this is the way that MySQL sends it and I haven't implemented any type conversions yet. Prepared statements return binary types so I've mapped some of these e.g. strings and numbers into their respective Scheme types.
I would strongly recommend using prepared statements rather than dynamic statements for performance reasons and also for their natural resistance to SQL injection attacks in web applications.
Once the library is finished, I'll create a persistence layer that builds on top of the prepared statements to give a framework similar to ActiveRecord in Rails or Hibernate in Java.
Tuesday, 21 April 2009
Gambit has also been proven to support tens of thousands of concurrent lightweight threads which helps make it an ideal platform for web applications. This coupled with it's rare ability to serialize continuations places it as a front runner for web applications in the 21st century.
Unfortunately there is little infrastructure there at present to support web applications ... or so I thought. It turns out that a tremendous amount of effort has gone into developing a framework called 'Black Hole' by Per Eckerdal and Mikael More. This is a continuations based framework that is built on top of a module system for Gambit. I look forward to seeing this being released in the very near future.
To support this effort, I have developed a pure Scheme MySQL interface for Gambit that is currently in the Gambit Dumping Grounds. The advantage of a pure Scheme implementation rather than a C binding is that it won't block your thousands of lightweight threads whilst your OS thread is stuck in C land.
Currently the implementation supports logging on with new and old authentication methods, switching schema's and DML and DDL using dynamic queries. I'm currently adding support for prepared statements which I would consider essential before building a persistence layer for Black Hole or anything else.
I'd love to do the same for Oracle but unfortunately Oracle don't publish the protocol for SQL*Net so this is not possible. Hopefully Gambit will devise a way to call C with an OS thread so as not to block the lightweight threads and then we can use the OCI interface.