Algorithmic Trading Group Forum

Algorithmic Trading Group Forum
https://www.algotradinggroup.com/cgi-bin/yabb2/YaBB.pl
Trading Strategies >> Software Infrastructure >> Ok, guys, help with a real project architecture!
https://www.algotradinggroup.com/cgi-bin/yabb2/YaBB.pl?num=1220365416

Message started by Yury R on 09/02/08 at 15:23:36

Title: Ok, guys, help with a real project architecture!
Post by Yury R on 09/02/08 at 15:23:36

Trying to enlighten the forum again...

This might seem stupid and a very dated idea. It is certainly below the level of at least some forum members. But it can be a good starting point for many.

There is a rather old concept of relative strength (RS) for equities. On any given day, If we take all stock dayly returns and order them in descending order we will be able to rank stocks from the highest return on that day to the lowest. Instead of considering nominal ranks we can split the whole ordered list into percentiles and, say, select the top 99 percentile (i.e. 1% of the highest returns on that day). That number (99) will be RS for the stock.

Now, for each stock, instead of having returns time series we can consider instead RS time series. Presumably, stocks don't jump from 99 to 1 on day-to day basis but rather have some pattern, which can be investigated.

So, one task would be to download all stocks history and transform it into RS history.

Next task would be to measure profitability of a simple portfolio that is distributed amongst top 1-2-3% and is rebalanced periodically (daily)

Another task is to take into account survivorship bias and measure its influence on the return on the above mentioned portfolio.

I am asking the forum members to contribute their ideas on how to implement a system that can do these 3 simple tasks. I only need software architecture ideas (hence, the location of the thread), namely:
1) where to take data from,
2) how to transform it
3) how to backtest the portfolio given the data obtained in 1) and 2)

I am not very interested in turn-key commercial solutions. I only want to build it myself from several pieces because it will allow me to do a research on the obtained data in my own unique way.

Comments like "it is all not very gifted" are welcome.

Title: Re: Ok, guys, help with a real project architecture!
Post by Co0olCat on 09/02/08 at 17:24:55

Off-shelf solution is to use PosgreSQL. You can write your own functions there. Data is from Yahoo or Google. In R you can find functions to download daily data. These functions create URL request and store CSV files to the disk. These files you can feed directly to the DB. Yahoo provides adjusted close prices to take into accounts splits and dividends.

After that you can perform whatever studies to investigate profitability and associated risk.

Title: Re: Ok, guys, help with a real project architecture!
Post by Yury R on 09/02/08 at 17:42:24

Ok, thanks!

Haven't discovered yet how to download quotes from Yahoo in R - should be very handy.

Why not MySQL? What is so special about Postgress? Never used it though....

The last task is still a problem - how to find/keep track of dead companies to measure the survovorship bias? Or should it normally be not too big - It is still a mistery to me.

Title: Re: Ok, guys, help with a real project architecture!
Post by kirppu on 09/03/08 at 09:45:51

Here is a sample java code to automatically download quotes from yahoo and convert it into a postgreSQL insert statement.

Title: Re: Ok, guys, help with a real project architecture!
Post by kirppu on 09/03/08 at 09:50:33

To automatically download and insert yahoo quotes in a postgreSQL table, use the previous java code and the following script:

#!/bin/bash

# 1) write the password for psql

export PGPASSWORD=xyz # edit you passoword

# 2) name of all yahoo symbols you want to download

a=( AAPL
ABB
^OMX
^RUT
^SPMIB
^SSMI)

COUNTER=0
while [ $COUNTER -lt ${#array

} ]; do
symbol=${a[$COUNTER]}
file=$symbol".csv"

#. 3. create an INSERT for each symbol
java Insert $file $symbol

#. 4. insert the timeseries in the database taq
psql taq -f $file &>0

# 5. remove the file
rm $file
let COUNTER=$COUNTER+1
done

Title: Re: Ok, guys, help with a real project architecture!
Post by kirppu on 09/03/08 at 09:52:02

Can somebody send us the equivalent in R?

Title: Re: Ok, guys, help with a real project architecture!
Post by Co0olCat on 09/03/08 at 10:06:51

To get data in R there is Quantmod package: http://www.quantmod.com

Check getSymbols.oanda(), getSymbols.google(), getSymbols.yahoo()

As for SQL you can use RODBC: http://cran.r-project.org/web/packages/RODBC/index.html

Note: To avoid performance issue I would suggest to import data to a DB through CSV file. Use PostgreSQL command COPY.

Title: Re: Ok, guys, help with a real project architecture!
Post by Yury R on 09/04/08 at 19:10:14

I would also download everthing into cvs file and then do bulk import into SQL server of choice.

I would prefer R-solution to Java+bash just to keep it simple at the begining.

Quantmod is good in a sense that it allows to limit the development scope - it can download from Yahoo through R, then save it in MySql or RODBC in R and retrieve it from MySQL or RODBC in R. Great!

But thanks to either of you!

Title: Re: Ok, guys, help with a real project architecture!
Post by Yury R on 09/06/08 at 20:01:18

Ok, this is my first attempt to download the data:
1. I went to finviz.com screener, selected "Average volume between 500k and 1m" got 686 stocks as a result, which I downloaded as "finviz.csv" file

2. This is the R script (using quantmod):

Code:

require("quantmod")
allstocks = read.csv("C:/data/stocks/finviz.csv", stringsAsFactors=F)
for(i in 1:length(allstocks$Ticker)) {
t = as.character(allstocks$Ticker[i])
stock = as.data.frame(getSymbols(t, src="yahoo", auto.assign=F))
names(stock) = c("Open", "High", "Low", "Close", "Volume", "Adjusted")
fname = paste(t, ".csv", sep="")
fname = paste("C:/data/stocks/", fname, sep="")
write.csv(stock, fname)
}

It took about 10 min to download the whole lot. Now I have got plenty of csv-files. And thinking of what to do next with them. To process, to SQL import them....