MQLmagazine.com

Guidelines for writing portfolio expert advisors in MQL5

Bogdan Caramalac, MQLmagazine sr.editor — Mon, 24 May 2010 21:29:43 +0000

[Versiunea romaneasca] [MQLmagazine.com in romana] [English edition]

Portfolio expert advisors, as well as the more complex multiasset expert advisors, were awaited by the retail trading community ever since limitations of the MT4 backtester became obvious, that is, about 4 – 5 years ago. The difference between multiasset and portfolio EAs is that multiasset EAs have an integrated strategy where each asset is a component, whether portfolio EAs replicate same strategy, with just some parameter differences, over more instruments. Of course, in the MT4 context, normal EAs could have been run on separate instruments and obtain portfolio effects, however backtesting could never have covered these portfolio effects because it was not possible to backtest on more than one instrument.

The main advantage of portfolying is the yield averaging while spreading the risk. The degree up to which this happens, is dependant on the overall portfolio correlation. The lesser overall correlated, the lesser nonsysthematic risk left uncovered. However, if you don’t pick the portfolio right, instead to balance your overall trading system, it will add extra instability.

A portfolio trading system must be able to:
- apply similar trading logics to a large number of assets;
- differentiate trading logics from one instrument to another;
- control exposure.

It requires a pretty large data structure to take make things manageable.

For instance, this one:

input double MarginUsagePerPosition=0.5;
 
int SymbolsCount;
 
struct IndicatorHandlersStruct
  {
   //...
  };
 
struct SystemParametersStruct
  {
   //...
  };
 
struct SymbolData
  {
    string Symbol;
    IndicatorHandlersStruct IndicatorHandlers;
    SystemParametersStruct SystemParameters;
  };
 
SymbolData SymbolsTable[300];

The first variable, MarginUsagePerPosition is a sort of asset allocation parameter, by controlling the maximal volume of each position. SymbolsCount tells how many symbols are used and it can be used for enumeration, while the big SymbolsTable[] will contain all the needed data, from symbols to handlers. For instance, the following structure will be set up to calculate 4 moving averages (two on M5 and two on H1) per each symbol:

struct IndicatorHandlersStruct
  {
   int MovingAveragesHandlers14[2];
   int MovingAveragesHandlers9[2];
  };
 
struct SymbolData
  {
    string Symbol;
    IndicatorHandlersStruct IndicatorHandlers;
    datetime LastBarTime;
  };
 
SymbolData SymbolsTable[300];

Before OnInit() we have to write a procedure to fill in the SymbolsTable[] with the indicator handlers. MakeIndicatorHandlers() is being called from OnInit() to separate the symbols setup from the indicators setup. While you can easily alter OnInit() to change the list, by adding, deleting or making an automated instrument selector, the MakeIndicatorHandlers() will remain static, with the purpose to fill the structure with the needed indicator handlers.

void MakeIndicatorHandlers()
  {
   datetime lastbar[1];
   for (int i=0;i<SymbolsCount;i++)
      {
       SymbolsTable[i].IndicatorHandlers.MovingAverageHandlers14[0]=iTEMA(SymbolsTable[i].Symbol,PERIOD_M5,14,0,PRICE_OPEN);
       SymbolsTable[i].IndicatorHandlers.MovingAverageHandlers9[0]=iTEMA(SymbolsTable[i].Symbol,PERIOD_M5,9,0,PRICE_OPEN);
       SymbolsTable[i].IndicatorHandlers.MovingAverageHandlers14[1]=iTEMA(SymbolsTable[i].Symbol,PERIOD_H1,14,0,PRICE_OPEN);
       SymbolsTable[i].IndicatorHandlers.MovingAverageHandlers9[1]=iTEMA(SymbolsTable[i].Symbol,PERIOD_H1,9,0,PRICE_OPEN);  
      }
   return;
  }

Within OnInit(), we set up the symbols table. For instance, we are making it to work on 3 forex pairs: EURUSD, USDCHF, GBPJPY:

int OnInit()
  {
   SymbolsTable[0].Symbol="EURUSD";
   SymbolsTable[1].Symbol="USDCHF";
   SymbolsTable[2].Symbol="GBPJPY";
   SymbolsCount=3;
   MakeIndicatorHandlers();
   return(0);
  }

And now OnTick():

void OnTick()
  {
   datetime datetime_array[1];
   for (int i=0;i<SymbolsCount;i++)
      {
       CopyTime(SymbolsTable[i].Symbol,PERIOD_M5,0,1,datetime_array);
       if (datetime_array[0]!=SymbolsTable[i].LastBarTime)
         {               
          TradeLogics(i);
          LastBarTime[i]=datetime_array[0];    
         }//if (datetime_array[0]!=SymbolsTable[i].LastBarTime)
      }//for (int i=0;i
   return;
  }

Now all the pieces are complete. OnTick() will check for each instrument if a new bar appeared and will call TradeLogics() for every instrument in the array, which will implement decision and trading.

We will not stop on TradeLogics() in this article, because TradeLogics() is EA specific. But a few principles should be guides while writing the TradeLogics(). The first division of the TradeLogics() is the asset class that it deals with. It can’t deal with equities the same way it deals with forex. Equities don’t run 24 hours a day 5 days a week. They have specific trading sessions and gaps from a day to another, not mentioning from a week to another. So this is the first division, the asset class. The second, is that it is better to separate the trading functions from the TradeLogics(). For instance, the TradeLogics() could just command how to adjust the position on a given instrument, and leave the trading function to a ManagePosition() procedure.
Of course, TradeLogics() can be in practice triggered by OnTrade() or OnChartEvent(), or even a CEP engine’s EventsCallback().

double UnitsToLots(double units,string symbol)
  {
  double dlotsize=SymbolInfoDouble(symbol,SYMBOL_TRADE_CONTRACT_SIZE);
  double mag=SymbolInfoDouble(symbol,SYMBOL_VOLUME_STEP);
  double mini=SymbolInfoDouble(symbol,SYMBOL_VOLUME_MIN);
  double big=1/mag;    
  double size=dlotsize/big;
  double dlots0,dlots1,back0,back1;
  dlots0=NormalizeDouble(units/size,0)*mag;
  dlots1=dlots0+mag;
  back0=dlots0*big*size;
  back1=dlots1*big*size;
  if (back1-units<units-back0)
    {
     if (dlots1<mini)
       return(mini);  
     else  
       return(dlots1);
    }
  else
    {
     if (dlots0==0)
        return(mini);
     else
       {
        if (dlots0<mini)
          return(mini);
        else
          return(dlots0);
       }//else if (dlots0==0)
    }//else if if (back1-units
  }

UnitsToLots() is a port of a function that I wrote in MQL4 times, and it was working at that time with MarketInfo(). UnitsToLots() will return the appropriate number of lots correspondant to a volume given in units. For instance it may answer 1.0 for 100000 units of EURUSD.

#define FLAT        0
#define LONG        1
#define SHORT       2
 
int GetPositionType(int asset_index)
  {
   string symbol=SymbolsTable[asset_index].Symbol;
   bool sel=PositionSelect(symbol);
   if (sel==false)
     return(FLAT);
   else
    {
     long p=PositionGetInteger(POSITION_TYPE);
     if (p==POSITION_TYPE_BUY)
       return(LONG);
     else
       return(SHORT);
    }
  }

This function is a simple proxy to get a straight position type value. The value for POSITION_TYPE_BUY is 0 in MQL5, and PositionInfoInteger() will answer 0 even if the position was not previously selected.

void ManagePosition(int asset_index,int operation,double forcevolume=0.00)
  {   
   MqlTradeRequest request;
   MqlTradeResult result;
   int p=GetPositionType(asset_index);
   double now_volume;
   double current_volume,v0;
   if (PositionSelect(SymbolsTable[asset_index].Symbol)==true)
     current_volume=PositionGetDouble(POSITION_VOLUME);
   else
     current_volume=0;
   v0=current_volume;   
   request.action=TRADE_ACTION_DEAL;
   request.symbol=SymbolsTable[asset_index].Symbol; 
   request.deviation=Slippage;
   request.type_filling=ORDER_FILLING_AON;
   request.type_time=ORDER_TIME_GTC;
   if (p==FLAT)
     {
      if (DoubleToString(forcevolume,2)=="0.00")
        request.volume=UnitsToLots( (MarginUsagePerPosition/100)*AccountInfoDouble(ACCOUNT_EQUITY)*AccountInfoInteger(ACCOUNT_LEVERAGE),SymbolsTable[asset_index].Symbol );
      else
        request.volume=forcevolume;
      if (operation==LONG)
        {
         request.type=ORDER_TYPE_BUY;
         request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_ASK),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));
        }
      if (operation==SHORT)
        {
         request.type=ORDER_TYPE_SELL;
         request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_BID),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));
        }
      if (operation==FLAT)
        return;
     }
   else//if (p==FLAT)
     {     
      if (p==LONG&&operation==SHORT)
        {
         if (DoubleToString(forcevolume,2)=="0.00")
           request.volume=current_volume+UnitsToLots( (MarginUsagePerPosition/100)*AccountInfoDouble(ACCOUNT_EQUITY)*AccountInfoInteger(ACCOUNT_LEVERAGE),SymbolsTable[asset_index].Symbol );
         else
           request.volume=current_volume+forcevolume;
         request.type=ORDER_TYPE_SELL;         
         request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_BID),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));         
        }
      if (p==SHORT&&operation==LONG)
        {
         if (DoubleToString(forcevolume,2)=="0.00")         
           request.volume=current_volume+UnitsToLots( (MarginUsagePerPosition/100)*AccountInfoDouble(ACCOUNT_EQUITY)*AccountInfoInteger(ACCOUNT_LEVERAGE),SymbolsTable[asset_index].Symbol );
         else
           request.volume=current_volume+forcevolume;
         request.type=ORDER_TYPE_BUY;
         request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_ASK),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));
        }
      if (p==LONG&&operation==FLAT)
        {       
         if (DoubleToString(forcevolume,2)=="0.00")
           request.volume=current_volume;
         else
           request.volume=forcevolume;
         request.type=ORDER_TYPE_SELL;         
         request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_BID),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));         
        }
      if (p==SHORT&&operation==FLAT)
        {
         if (DoubleToString(forcevolume,2)=="0.00")           
           request.volume=current_volume;
         else
           request.volume=forcevolume;
         request.type=ORDER_TYPE_BUY;         
         request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_ASK),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));         
        } 
      if (p==LONG&&operation==LONG)//adjustment of the present LONG position
        {
         if (DoubleToString(forcevolume,2)=="0.00")
           request.volume=UnitsToLots( (MarginUsagePerPosition/100)*AccountInfoDouble(ACCOUNT_EQUITY)*AccountInfoInteger(ACCOUNT_LEVERAGE),SymbolsTable[asset_index].Symbol )-current_volume;
         else
           request.volume=forcevolume-current_volume;
         if (NormalizeDouble(request.volume,2)>0.00)
           {         
            request.type=ORDER_TYPE_BUY;
            request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_ASK),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));
           }
         else //we close the difference
           {
            request.volume=MathAbs(request.volume);
            request.type=ORDER_TYPE_SELL;         
            request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_BID),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));         
           }
        }        
      if (p==SHORT&&operation==SHORT)//adjustment of the present SHORT position
        {
         if (DoubleToString(forcevolume,2)=="0.00")
           request.volume=UnitsToLots( (MarginUsagePerPosition/100)*AccountInfoDouble(ACCOUNT_EQUITY)*AccountInfoInteger(ACCOUNT_LEVERAGE),SymbolsTable[asset_index].Symbol )-current_volume;
         else
           request.volume=forcevolume-current_volume;
         if (NormalizeDouble(request.volume,2)>0.00)
           {
            request.type=ORDER_TYPE_SELL;         
            request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_BID),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));                  
           }
         else
           {
            request.volume=MathAbs(request.volume);
            request.type=ORDER_TYPE_BUY;
            request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_ASK),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));            
           }
        }        
     }//if (p==FLAT)]  
   if (DoubleToString(request.volume,2)!="0.00")
     {
      OrderSend(request,result);     
      PositionSetSLTP(StopLoss,TakeProfit);
     }     
  }

What does ManagePosition() do ? It opens a position on the given asset by index, upon request, if it is flattened before ; or it reverts the current position, by recalculating the new lot size (calculates new lot size with the formula, adds to current volume and reverses operation). It can also add or cut from the current position. The forcevolume parameter can be used to force a given volume into a trade, bypassing calculus.
How about setting a stop loss ? Nothing easier:

void PositionSetSLTP(int asset_index,int sl,int tp)
  {
   bool todosl, todotp;
   MqlTradeRequest request;
   MqlTradeResult result;   
   double price=0.00;
   int p=GetPositionType(asset_index);
   if (p==LONG||p==SHORT)
     price=PositionGetDouble(POSITION_PRICE_OPEN);
   request.action=TRADE_ACTION_SLTP;
   request.symbol=SymbolsTable[asset_index].Symbol;   
   todosl=false;
   todotp=false;
   if (DoubleToString(sl,4)!="0.0000")
     {
      todosl=true;      
      if (p==LONG)
        request.sl=NormalizeDouble(price-sl*SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_POINT),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));
      if (p==SHORT)
        request.sl=NormalizeDouble(price+sl*SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_POINT),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));
     }
   if (DoubleToString(tp,4)!="0.0000")
     {      
      todotp=true;
      if (p==LONG)
        request.tp=NormalizeDouble(price+tp*SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_POINT),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));
      if (p==SHORT)
        request.tp=NormalizeDouble(price-tp*SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_POINT),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));
     }
   request.deviation=Slippage;   
   if (todosl==true||todotp==true)
     OrderSend(request,result);
  }

Note that the OrderSend() calls are not followed by an analysis of the result. We didn’t dwelve into retcodes because some of them are not completely clear to us.

And finally OnDeinit(), which destroys the indicator handlers.

void OnDeinit(const int reason)
  {
   for (int i=0;i<SymbolsCount;i++)
      {   
       IndicatorRelease(SymbolsTable[i].IndicatorHandlers.MovingAverageHandlers14[0]);
       IndicatorRelease(SymbolsTable[i].IndicatorHandlers.MovingAverageHandlers9[0]);
       IndicatorRelease(SymbolsTable[i].IndicatorHandlers.MovingAverageHandlers14[1]);
       IndicatorRelease(SymbolsTable[i].IndicatorHandlers.MovingAverageHandlers9[1]);
      }
   return;
  }

MQL5 : A portfolio moving average sample expert

Bogdan Caramalac, MQLmagazine sr.editor — Mon, 24 May 2010 21:29:26 +0000

[Versiunea romaneasca] [MQLmagazine.com in romana] [English edition]

I wrote this expert advisor to as a full demonstration on writing portfolio expert advisors, following the principles presented in Guidelines for writing portfolio expert advisors in MQL5.

The expert will work on the chosen timeframe, only when a new bar forms up on every participating instrument. Upon checking moving average intersections, it will manage positions. There is also the option to filter by volatility (I repaired the BBoverSTD indicator from the Volatility analysis : bridging the gap from volatility forecasting to price forecasting article).

//+------------------------------------------------------------------+
//|                                                  PortfolioMA.mq5 |
//|                                       Copyright Bogdan Caramalac |
//|                                           http://mqlmagazine.com |
//+------------------------------------------------------------------+
#property copyright "Copyright Bogdan Caramalac"
#property link      "http://mqlmagazine.com"
#property version   "1.00"
 
#define FLAT        0
#define LONG        1
#define SHORT       2
 
input int Slippage=30;
input double MarginUsagePerPosition=0.5;
input bool VolatilityFilter=false;
input int StopLoss=300;
input int TakeProfit=1000;
 
input int SlowMALength=14;
input int FastMALength=9;

This section defines the constants for working with positions (FLAT,LONG,SHORT) and the parameters of the system.

int SymbolsCount;
 
struct IndicatorHandlersStruct
  {
   int SlowMovingAverageHandler;
   int FastMovingAverageHandler;
   int BBoverSTDHandler;
  };
 
struct SymbolData
  {
    string Symbol;
    IndicatorHandlersStruct IndicatorHandlers;
    datetime LastBarTime;    
  };
 
SymbolData SymbolsTable[30];

This section defines the data structure holding the symbols and indicator handlers.

void MakeIndicatorHandlers()
  {
   datetime datetime_array[1];
   int error1,error2,error3;
   for (int i=0;i<SymbolsCount;i++)
      {
       error1=0;
       error2=0;
       error3=0;
       SymbolsTable[i].IndicatorHandlers.SlowMovingAverageHandler=iMA(SymbolsTable[i].Symbol,Period(),SlowMALength,0,MODE_SMA,PRICE_OPEN);
       Print("SymbolsTable[",i,"].IndicatorHandlers.SlowMovingAverageHandler=",SymbolsTable[i].IndicatorHandlers.SlowMovingAverageHandler);
       if (SymbolsTable[i].IndicatorHandlers.SlowMovingAverageHandler<0)
         { 
          Print("Invalid handle.");
          error1=GetLastError();
          Print("Error ",error1," while initializing iMA(",SlowMALength,") for ",SymbolsTable[i].Symbol);
         }
       SymbolsTable[i].IndicatorHandlers.FastMovingAverageHandler=iMA(SymbolsTable[i].Symbol,Period(),FastMALength,0,MODE_SMA,PRICE_OPEN);
       Print("SymbolsTable[",i,"].IndicatorHandlers.FastMovingAverageHandler=",SymbolsTable[i].IndicatorHandlers.FastMovingAverageHandler);
       if (SymbolsTable[i].IndicatorHandlers.FastMovingAverageHandler<0)
         {
          Print("Invalid handle.");
          error2=GetLastError();
          Print("Error ",error2," while initializing iMA(",FastMALength,") for ",SymbolsTable[i].Symbol);
         }
       SymbolsTable[i].IndicatorHandlers.BBoverSTDHandler=iCustom(SymbolsTable[i].Symbol,Period(),"BBoverSTD",20,10,1.5,"SMA",2,0);
       Print("SymbolsTable[",i,"].IndicatorHandlers.BBoverSTDHandler=",SymbolsTable[i].IndicatorHandlers.BBoverSTDHandler);
       if (SymbolsTable[i].IndicatorHandlers.BBoverSTDHandler<0)
         { 
          Print("Invalid handle.");
          error3=GetLastError();
          Print("Error ",error3," while initializing BBoverSTD for ",SymbolsTable[i].Symbol);
         }                   
      }
   return;
  }

This is the MakeIndicatorHandlers() routine tailored for this EA.

Now follow UnitsToLots(), GetPositionType(), PositionSetSLTP() and ManagePosition() that are the ones presented in the Guidelines for writing portfolio expert advisors in MQL5.

string MASignal(int asset_index)
  {
   double maslowarray[1];
   double mafastarray[1];
   double mafast,maslow;
   datetime t;
   int copycount1,copycount2;
   int error1=0;
   int error2=0;
   string signal;
   copycount1=CopyBuffer(SymbolsTable[asset_index].IndicatorHandlers.SlowMovingAverageHandler,0,0,1,maslowarray);
   if (copycount1==-1)
     {
      Print("Nothing copied (1). Retrieving error.");
      error1=GetLastError();
     }
   copycount2=CopyBuffer(SymbolsTable[asset_index].IndicatorHandlers.FastMovingAverageHandler,0,0,1,mafastarray);
   if (copycount2==-1)
     {
      Print("Nothing copied (2). Retrieving error.");
      error2=GetLastError();
     }
   maslow=NormalizeDouble(maslowarray[0],SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));
   mafast=NormalizeDouble(mafastarray[0],SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));   
   t=TimeCurrent();
   if (maslow<mafast)
     signal="L";
   if (maslow==mafast)
     signal="-";
   if (maslow>mafast)
     signal="S";
   return(signal);
  }

This returns the moving average signal for the requested asset. Signal can be “L”, “S” or “-” .

void TradeLogics(int asset_index)
  {
   string s;
   double stddev[1];
   double bbhi[1];
   double bblo[1];
   int copycount;
   s=MASignal(asset_index);   
   if (VolatilityFilter==true)
     {
      copycount=CopyBuffer(SymbolsTable[asset_index].IndicatorHandlers.BBoverSTDHandler,0,0,1,stddev);
      copycount=CopyBuffer(SymbolsTable[asset_index].IndicatorHandlers.BBoverSTDHandler,2,0,1,bbhi);
     }
   if (GetPositionType(asset_index)==FLAT)
     {      
      if (s=="L")
        {
         if (VolatilityFilter==true)
           {
            if (stddev[0]>bbhi[0])
              ManagePosition(asset_index,LONG);
           }
         else
           ManagePosition(asset_index,LONG);
        }
      if (s=="S")
        {
         if (VolatilityFilter==true)
           {
            if (stddev[0]>bbhi[0])
              ManagePosition(asset_index,SHORT);
           }
         else
           ManagePosition(asset_index,SHORT);
        }
     }//else if (GetPositionType(asset_index)==FLAT)
   else//position is not flat
     {
      if (GetPositionType(asset_index)==LONG)
        {
         if (VolatilityFilter==true)
           {
            if (s=="S"||stddev[0]<bbhi[0])
              ManagePosition(asset_index,FLAT);
           }           
         else
           {
            if (s=="S")
              ManagePosition(asset_index,FLAT);           
           }
        }
      if (GetPositionType(asset_index)==SHORT)
        {
         if (VolatilityFilter==true)
           {
            if (s=="L"||stddev[0]<bbhi[0])
              ManagePosition(asset_index,FLAT);
           }
         else
           {
            if (s=="L")
              ManagePosition(asset_index,FLAT);           
           }        
        }
     }           
   return;
  }

This is the TradeLogics(). First it checks if VolatilityFilter is enabled, and only in this case calls the BBoverSTD indicator, to save time for the situation the filter is disabled. Then it tests if position is flat.
If it is, goes long or short according to the MASignal() , eventually filtered by the BBoverSTD values, if VolatilityFilter is enabled. If the position is long or short already, it checks only for contrarian signals (of course, same filtering). If contrarian signal comes, position is flattened, otherwise there is no adaptation of position to the new equity size (although the ManagePosition() is able to do it, but not asked to do so).

//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+
int OnInit()
  {
   string crtsymbol;
   SymbolsTable[0].Symbol="EURUSD";
   SymbolsTable[1].Symbol="USDJPY";
   SymbolsTable[2].Symbol="GBPUSD";
   SymbolsTable[3].Symbol="USDCHF";
   SymbolsCount=1;
   MakeIndicatorHandlers();
   return(0);
  }

This is the OnInit(). It is already preset to work with four currency pairs. If SymbolsCount remains 1 , only EURUSD will be traded. If you set it to 4, all four pairs will be traded. You can alter the SymbolsTable[] however you wish (problems may arise however at the lot calculus while trading CFDs). OnInit() must end with a call to MakeIndicatorHandlers() to get the indicator handlers ready for usage.

//+------------------------------------------------------------------+
//| Expert tick function                                             |
//+------------------------------------------------------------------+
void OnTick()
  {
   datetime datetime_array[1];
   for (int i=0;i<SymbolsCount;i++)
      {
       CopyTime(SymbolsTable[i].Symbol,Period(),0,1,datetime_array);
       if (datetime_array[0]!=SymbolsTable[i].LastBarTime)
         {
          //Print(SymbolsTable[i].Symbol," :: ",TimeToString(SymbolsTable[i].LastBarTime,TIME_DATE)," ",TimeToString(SymbolsTable[i].LastBarTime,TIME_MINUTES),"  vs. ",TimeToString(datetime_array[0],TIME_DATE)," ",TimeToString(datetime_array[0],TIME_SECONDS) );
          TradeLogics(i);
          SymbolsTable[i].LastBarTime=datetime_array[0];          
         }//if (datetime_array[0]!=LastBarOccurred[i])
      }//for (int i=0;i
   return;
  }

This is the OnTick(). The OnTick() contains the symbol loop, and requires the time of the last bar of the given timeframe on the current symbol given by the cycle. If this time is different than remembered time, it means a new bar appeared on that symbol chart. Then TradeLogics() is called for decision and subsequent call to ManagePosition().

//+------------------------------------------------------------------+
//| Expert deinitialization function                                 |
//+------------------------------------------------------------------+
void OnDeinit(const int reason)
  {
   for (int i=0;i<SymbolsCount;i++)
      {   
       IndicatorRelease(SymbolsTable[i].IndicatorHandlers.SlowMovingAverageHandler);
       IndicatorRelease(SymbolsTable[i].IndicatorHandlers.FastMovingAverageHandler);
       IndicatorRelease(SymbolsTable[i].IndicatorHandlers.BBoverSTDHandler);
      }
   return;
  }

This is the OnDeinit(). It releases indicator handlers. Should OnInit() be run again, it will make other handlers. It is not completely necessary to do this, however wasting memory with new handlers every time OnInit() runs is not clean programming.

The following backtests were made with no StopLoss and TakeProfit set, also with VolatilityFilter disabled.

As we can see, portfolying was not a good choice. It added extra instability and deeper drawdowns. However, now we are dealing with trading systems, while portfolio management theory supposes a generally long or out strategy. This why, in financial theory, negative correlations would drive risks down, because pluses on some assets would compensate with minuses on other assets. But, what we have now is a portfolio of trading systems. We can think a trading system as an equity function of instruments:

 equity = trading system (instrument)

On two perfectly negatively correlated assets, the trading system function would output two positively correlated equity series. A rigorous approach would ask that a Markowitz or CAPM selection over trading systems to be done with based on trading system outputs rather than market data.

See also the article about trading systems virtualization article, which is analysing the problem from another point of view.

File link:
PortfolioMA.mq5

Trading systems virtualization to achieve equity line fitting – a myth ?

Bogdan Caramalac, MQLmagazine sr.editor — Mon, 24 May 2010 21:29:07 +0000

[Versiunea romaneasca] [MQLmagazine.com in romana] [English edition]

Did you ever think that you’d like to have a moving average over the equity of a trading system, so you can turn it on and off ? This would be called equity line fitting.

The frustrating aspect is that once you start doing this from the trading system that you want to curve it its equity line, you cannot control the process anymore, as the indicators of the equity line will report differently once you hit the switch.

So, what you have to do is to let the system paper trade somehow in parallel to your main trading system, and copy the trades in your main trading system when the switch is on, ignore paper trades when the switch is off.

This can be done by virtualization. Meaningly, you have to write a paper trade position management system, like a virtual account and output its “equity” to the main trading system, which will curve-fit this equity and decide whether to replicate or not the paper system’s trades.

Thus, we begin writing such a system, in file which we’ll call TradeVirtualization.mqh .

//+------------------------------------------------------------------+
//|                                          TradeVirtualization.mqh |
//|                                       Copyright Bogdan Caramalac |
//|                                           http://mqlmagazine.com |
//+------------------------------------------------------------------+
#property copyright "Copyright Bogdan Caramalac"
#property link      "http://mqlmagazine.com"
#property version "1.0"
 
#define FLAT        0
#define LONG        1
#define SHORT       2
 
struct VirtualSymbolData
  {
    string Symbol;    
    double position;
    double price;
    double sl;
    double tp;
    double marked_price;    
  };
 
struct VirtualEquityData
  {
   datetime opentime,closetime;
   double open,high,low,close;  
  };

We begin with the regular defines for position management, followed by two structures. The first is intended to store symbol data in, providing symbol, position (positive – long, negative – short), price (price of initiation for the position), marked price (last price at which the position was marked to market). The second is the intended to store virtual equity data : open and close times of an equity bar ; open, high, low, close for the equity within interval. Now it’s time for the VirtualAccount class.

class VirtualAccount
  {   
   public:
   int VirtualSymbolsCount;    
   VirtualSymbolData VirtualSymbolsTable[30]; 
   VirtualEquityData VirtualEquity[500];
   double VirtualLastAccountEquity;  
   int VirtualLastEquityPos;
   uint VirtualEquityBarSpan;
 
   void VirtualAccountSet(double Equity,uint TimeSpan)   
       {
        datetime tc=TimeCurrent();
        VirtualEquity[0].opentime=tc;
        VirtualEquity[0].open=Equity;
        VirtualEquity[0].high=Equity;
        VirtualEquity[0].low=Equity;
        VirtualEquity[0].close=Equity;
        VirtualEquity[0].closetime=tc;
        VirtualLastAccountEquity=VirtualEquity[0].close;
        VirtualEquityBarSpan=TimeSpan;
        VirtualLastEquityPos=0;
        return;
       }  
 
    int VirtualGetPositionType(int asset_index); 
    double VirtualGetPositionSize(int asset_index);
    void VirtualPositionSetSL(double asset_index,double sl);
    void VirtualPositionSetTP(double asset_index,double tp);
    void VirtualMarkToMarket(int asset_index);
    void VirtualChangePosition(int asset_index,int direction,double price,double lotsize);
    private:
    void VirtualAddNewEquity(double equity);
 
    public:
    VirtualAccount() 
       {
        VirtualAccountSet(AccountInfoDouble(ACCOUNT_EQUITY),0);
       };      
  };

The class contains an array of VirtualSymbolsData and its count, and an array with VirtualEquityData. Other key parameters are VirtualLastEquityPos and VirtualEquityBarSpan (the time interval of a bar, in seconds). VirtualLastAccountEquity is just a proxy for retrieving last recorded equity.

int VirtualAccount::VirtualGetPositionType(int asset_index)     
   {
    if (DoubleToString(NormalizeDouble(VirtualSymbolsTable[asset_index].position,2),2)=="0.00")
      return(FLAT);
    if (VirtualSymbolsTable[asset_index].position>0.00)
      return(LONG);
    if (VirtualSymbolsTable[asset_index].position<0.00)
      return(SHORT); 
    return(0);       
   }
 
double VirtualAccount::VirtualGetPositionSize(int asset_index)     
   {
    if (DoubleToString(NormalizeDouble(VirtualSymbolsTable[asset_index].position,2),2)=="0.00")
      return(0.00);
    return(NormalizeDouble(MathAbs(VirtualSymbolsTable[asset_index].position),2));       
   }       
 
 void VirtualAccount::VirtualPositionSetSL(double asset_index,double sl)
     {
      VirtualSymbolsTable[asset_index].sl=sl;
     }
 
 void VirtualAccount::VirtualPositionSetTP(double asset_index,double tp)
     {
      VirtualSymbolsTable[asset_index].tp=tp;
     }

VirtualGetPositionType() and VirtualGetPositionSize() are pure informational. The position size is always positive, like the POSITION_VOLUME requests to MQL5’s PositionInfoDouble(). The other two functions do what their name says they do.

void VirtualAccount::VirtualAddNewEquity(double equity)
    {
     datetime tc=TimeCurrent();
     uint diff;
     diff=tc-VirtualEquity[VirtualLastEquityPos].opentime;
     if (diff<=VirtualEquityBarSpan)
       {
        if (VirtualEquity[VirtualLastEquityPos].high<equity)
          VirtualEquity[VirtualLastEquityPos].high=equity;
        if (VirtualEquity[VirtualLastEquityPos].low>equity)
          VirtualEquity[VirtualLastEquityPos].low=equity;
        VirtualEquity[VirtualLastEquityPos].close=equity;
        diff=tc-VirtualEquity[VirtualLastEquityPos].closetime;           
        VirtualEquity[VirtualLastEquityPos].closetime=tc;           
        VirtualLastAccountEquity=equity;
        return;
       }
     if (VirtualLastEquityPos!=ArrayRange(VirtualEquity,0)-1)
       {           
        VirtualLastEquityPos++;
        VirtualEquity[VirtualLastEquityPos].opentime=tc;
        VirtualEquity[VirtualLastEquityPos].open=equity;
        VirtualEquity[VirtualLastEquityPos].high=equity;
        VirtualEquity[VirtualLastEquityPos].low=equity;
        VirtualEquity[VirtualLastEquityPos].close=equity;           
        VirtualEquity[VirtualLastEquityPos].closetime=tc;
        VirtualLastAccountEquity=equity;
        return;
       }
     else
       {
        for (int i=0;i<ArrayRange(VirtualEquity,0)-1;i++)
           VirtualEquity[i]=VirtualEquity[i+1];                  
        VirtualEquity[VirtualLastEquityPos].opentime=tc;
        VirtualEquity[VirtualLastEquityPos].open=equity;
        VirtualEquity[VirtualLastEquityPos].high=equity;
        VirtualEquity[VirtualLastEquityPos].low=equity;
        VirtualEquity[VirtualLastEquityPos].close=equity;           
        VirtualEquity[VirtualLastEquityPos].closetime=tc;
        VirtualLastAccountEquity=equity;
        return;
       }        
    }

At the beginning, the class is automatically set up with an equity equal to the account equity it runs on and a timespan of 0. The user may set the account to have a different startup equity and timespan. At every tick, VirtualMarkToMarket(-1) has to be called to ensure that all positions are marked to market with the new tick data. After each mark to market (either individual, with a positive parameter, or overall, with a negative parameter), a new equity value is calculated. Now the recording in the VirtualEquity[] begins from 0. If the timespan is not 0, and the new tick comes within the timespan from last equity bar opening time, the new tick is recorded on that bar, on the close field, stamping also the closetime field. High and low per equity bar is recalculated. If the timespan is 0, or if the new equity comes too late, it enters on the following bar, and so on, up to the maximum subscript (499). When it is to enter on 500, array is slided to the left one position and bar 499 starts to be recorded again. VirtualAddNewEquity() is called only from this class. Thus, it was designated to be a private method.

void VirtualAccount::VirtualMarkToMarket(int asset_index) //marks a new equity value
    {
     double fluctuation=0.00;
     double rprice,new_equity;
     int p;
     if (asset_index<0)//mark all
       {
        for (int i=0;i<VirtualSymbolsCount;i++)
           {
            if (DoubleToString(VirtualSymbolsTable[i].position,2)=="0.00")
              continue;
            if (NormalizeDouble(VirtualSymbolsTable[i].position,2)>0.00)
              rprice=NormalizeDouble(SymbolInfoDouble(VirtualSymbolsTable[i].Symbol,SYMBOL_BID),SymbolInfoInteger(VirtualSymbolsTable[i].Symbol,SYMBOL_DIGITS));
            if (NormalizeDouble(VirtualSymbolsTable[i].position,2)<0.00)
              rprice=NormalizeDouble(SymbolInfoDouble(VirtualSymbolsTable[i].Symbol,SYMBOL_ASK),SymbolInfoInteger(VirtualSymbolsTable[i].Symbol,SYMBOL_DIGITS));                
            fluctuation=fluctuation+MathAbs(VirtualSymbolsTable[i].position)*SymbolInfoDouble(VirtualSymbolsTable[i].Symbol,SYMBOL_TRADE_CONTRACT_SIZE)*(rprice-VirtualSymbolsTable[i].marked_price);               
            VirtualSymbolsTable[i].marked_price=rprice;
            if (DoubleToString(VirtualSymbolsTable[i].sl,5)!="0.00000")
              {
               if (VirtualSymbolsTable[i].position>0.00&&rprice<=VirtualSymbolsTable[i].sl)
                 {
                  VirtualSymbolsTable[i].position=0.00;                     
                  VirtualSymbolsTable[i].tp=0.00;
                  VirtualSymbolsTable[i].sl=0.00;
                 }
               if (VirtualSymbolsTable[i].position<0.00&&rprice>=VirtualSymbolsTable[i].sl)
                 {
                  VirtualSymbolsTable[i].position=0.00;                     
                  VirtualSymbolsTable[i].tp=0.00;
                  VirtualSymbolsTable[i].sl=0.00;
                 }                    
              }
            if (DoubleToString(VirtualSymbolsTable[i].tp,5)!="0.00000")
              {
               if (VirtualSymbolsTable[i].position>0.00&&rprice>=VirtualSymbolsTable[i].tp)
                 {
                  VirtualSymbolsTable[i].position=0.00;                     
                  VirtualSymbolsTable[i].tp=0.00;
                  VirtualSymbolsTable[i].sl=0.00;
                 }
               if (VirtualSymbolsTable[i].position<0.00&&rprice<=VirtualSymbolsTable[i].tp)
                 {
                  VirtualSymbolsTable[i].position=0.00;                     
                  VirtualSymbolsTable[i].tp=0.00;
                  VirtualSymbolsTable[i].sl=0.00;
                 }                    
              }                         
           }
        new_equity=VirtualEquity[VirtualLastEquityPos].close+fluctuation;
        VirtualAddNewEquity(new_equity);  
       }                      
     else
       {
        if (DoubleToString(VirtualSymbolsTable[asset_index].position,2)!="0.00")
          {
           if (NormalizeDouble(VirtualSymbolsTable[asset_index].position,2)>0.00)
             {
              p=LONG;
              rprice=NormalizeDouble(SymbolInfoDouble(VirtualSymbolsTable[asset_index].Symbol,SYMBOL_BID),SymbolInfoInteger(VirtualSymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));
             }
           if (NormalizeDouble(VirtualSymbolsTable[asset_index].position,2)<0.00)
             {
              p=SHORT;
              rprice=NormalizeDouble(SymbolInfoDouble(VirtualSymbolsTable[asset_index].Symbol,SYMBOL_ASK),SymbolInfoInteger(VirtualSymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));                
             }
           //if (p==LONG)
           //  Print("Market marking @ ",DoubleToString(rprice,SymbolInfoInteger(VirtualSymbolsTable[asset_index].Symbol,SYMBOL_DIGITS))," for ",VirtualSymbolsTable[asset_index].Symbol," ",DoubleToString(VirtualSymbolsTable[asset_index].position,2)," LONG @ ",DoubleToString(VirtualSymbolsTable[asset_index].price,SymbolInfoInteger(VirtualSymbolsTable[asset_index].Symbol,SYMBOL_DIGITS)),"last mark @ ",DoubleToString(VirtualSymbolsTable[asset_index].marked_price,SymbolInfoInteger(VirtualSymbolsTable[asset_index].Symbol,SYMBOL_DIGITS))," -> ",DoubleToString(new_equity,2)," by fl. ",DoubleToString(fluctuation,2));
           //if (p==SHORT)
           //  Print("Market marking @ ",DoubleToString(rprice,SymbolInfoInteger(VirtualSymbolsTable[asset_index].Symbol,SYMBOL_DIGITS))," for ",VirtualSymbolsTable[asset_index].Symbol," ",DoubleToString(VirtualSymbolsTable[asset_index].position,2)," SHORT @ ",DoubleToString(VirtualSymbolsTable[asset_index].price,SymbolInfoInteger(VirtualSymbolsTable[asset_index].Symbol,SYMBOL_DIGITS)),"last mark @ ",DoubleToString(VirtualSymbolsTable[asset_index].marked_price,SymbolInfoInteger(VirtualSymbolsTable[asset_index].Symbol,SYMBOL_DIGITS))," -> ",DoubleToString(new_equity,2)," by fl. ",DoubleToString(fluctuation,2));                
           fluctuation=fluctuation+MathAbs(VirtualSymbolsTable[asset_index].position)*SymbolInfoDouble(VirtualSymbolsTable[asset_index].Symbol,SYMBOL_TRADE_CONTRACT_SIZE)*(rprice-VirtualSymbolsTable[asset_index].marked_price);               
           VirtualSymbolsTable[asset_index].marked_price=rprice;
           if (DoubleToString(VirtualSymbolsTable[asset_index].sl,5)!="0.00000")
             {
              if (VirtualSymbolsTable[asset_index].position>0.00&&rprice<=VirtualSymbolsTable[asset_index].sl)
                {
                 VirtualSymbolsTable[asset_index].position=0.00;                     
                 VirtualSymbolsTable[asset_index].tp=0.00;
                 VirtualSymbolsTable[asset_index].sl=0.00;
                }
              if (VirtualSymbolsTable[asset_index].position<0.00&&rprice>=VirtualSymbolsTable[asset_index].sl)
                {
                 VirtualSymbolsTable[asset_index].position=0.00;                     
                 VirtualSymbolsTable[asset_index].tp=0.00;
                 VirtualSymbolsTable[asset_index].sl=0.00;
                }                    
             }
           if (DoubleToString(VirtualSymbolsTable[asset_index].tp,5)!="0.00000")
             {
              if (VirtualSymbolsTable[asset_index].position>0.00&&rprice>=VirtualSymbolsTable[asset_index].tp)
                {
                 VirtualSymbolsTable[asset_index].position=0.00;                     
                 VirtualSymbolsTable[asset_index].tp=0.00;
                 VirtualSymbolsTable[asset_index].sl=0.00;
                }
              if (VirtualSymbolsTable[asset_index].position<0.00&&rprice<=VirtualSymbolsTable[asset_index].tp)
                {
                 VirtualSymbolsTable[asset_index].position=0.00;                     
                 VirtualSymbolsTable[asset_index].tp=0.00;
                 VirtualSymbolsTable[asset_index].sl=0.00;
                }                    
             }                                       
          }
        new_equity=VirtualEquity[VirtualLastEquityPos].close+fluctuation;
        VirtualAddNewEquity(new_equity);             
       }
     return;
    }

VirtualMarkToMarket() is the heart of the entire class. It can be called for a position or for all positions. It marks positions to the market, calculating the account equity. For each position it calculates the fluctuation in equity from the last marked price to the current market price, which becomes marked price. For the same position, the procedure may clear the stop loss and take profits once these are taken. The user has to call VirtualMarkToMarket(-1) at every tick to mark positions to market.

void VirtualAccount::VirtualChangePosition(int asset_index,int direction,double price,double lotsize) //direction is ORDER_TYPE_BUY or ORDER_TYPE_SELL
    {
     int p=VirtualGetPositionType(asset_index);
     double rprice,fluctuation,new_equity;
     double oldposition;            
     if (p==FLAT)
       {
        VirtualSymbolsTable[asset_index].price=price;
        VirtualSymbolsTable[asset_index].marked_price=price;
        VirtualSymbolsTable[asset_index].sl=0.00000;
        VirtualSymbolsTable[asset_index].tp=0.00000;
        if (direction==ORDER_TYPE_BUY)             
          VirtualSymbolsTable[asset_index].position=NormalizeDouble(lotsize,2);
        if (direction==ORDER_TYPE_SELL)
          VirtualSymbolsTable[asset_index].position=NormalizeDouble(-lotsize,2);
        //Print("Position set to ",VirtualSymbolsTable[asset_index].position," @ price ",DoubleToString(VirtualSymbolsTable[asset_index].price,SymbolInfoInteger(VirtualSymbolsTable[asset_index].Symbol,SYMBOL_DIGITS) ) );
       }
     else
       {
        oldposition=VirtualSymbolsTable[asset_index].position;
        //local mark to market
        VirtualMarkToMarket(asset_index);           
        //position update
        VirtualSymbolsTable[asset_index].price=price;           
        if (direction==ORDER_TYPE_BUY)
          {
           VirtualSymbolsTable[asset_index].position=NormalizeDouble(VirtualSymbolsTable[asset_index].position+lotsize,2);
           if (oldposition<0.00&&VirtualSymbolsTable[asset_index].position>0.00||DoubleToString(VirtualSymbolsTable[asset_index].position,5)=="0.00000")
             {
              VirtualSymbolsTable[asset_index].sl=0.00000;
              VirtualSymbolsTable[asset_index].tp=0.00000;
             }
          }
        if (direction==ORDER_TYPE_SELL)
          {
           VirtualSymbolsTable[asset_index].position=NormalizeDouble(VirtualSymbolsTable[asset_index].position-lotsize,2);
           if (oldposition>0.00&&VirtualSymbolsTable[asset_index].position<0.00||DoubleToString(VirtualSymbolsTable[asset_index].position,5)=="0.00000")
             {
              VirtualSymbolsTable[asset_index].sl=0.00000;
              VirtualSymbolsTable[asset_index].tp=0.00000;
             }
          }              
        //Print("Position set to ",VirtualSymbolsTable[asset_index].position," @ price ",DoubleToString(VirtualSymbolsTable[asset_index].price,SymbolInfoInteger(VirtualSymbolsTable[asset_index].Symbol,SYMBOL_DIGITS) ) );
       }
    return;
   }

To make a trade, you have to use VirtualChangePosition() which will open a position with the given direction , price and lot size on the specified asset index. This opening is similar to a deal. Thus, only ORDER_TYPE_BUY and ORDER_TYPE_SELL are supported for operation. No pending or StopLimit orders. The function only sets or modifies positions. It also clears the previous stop loss and take profit if a position reverses sign.

Now that the class is done, we can virtualize the portfolio moving average expert from the MQL5 : A portfolio moving average sample expert article in a new file, called VirtualPortfolioMA.mq5 .

//+------------------------------------------------------------------+
//|                                           VirtualPortfolioMA.mq5 |
//|                                       Copyright Bogdan Caramalac |
//|                                           http://mqlmagazine.com |
//+------------------------------------------------------------------+
#property copyright "Copyright Bogdan Caramalac"
#property link      "http://mqlmagazine.com"
#property version   "1.00"
 
#define FLAT        0
#define LONG        1
#define SHORT       2
 
input int Slippage=30;
input double MarginUsagePerPosition=0.5;
input bool VolatilityFilter=false;
input int VirtualStopLoss=300;
input int VirtualTakeProfit=1000;
input int SlowMALength=14;
input int FastMALength=9;
 
input int EquitySlowMAPeriod=14;
input int EquityFastMAPeriod=9;
input int RealStopLoss=300;
input int RealTakeProfit=1000;

The variables are quite the same, with the distinction that now stop loss , take profit and moving average periods are split in two : for the virtual account and for the account the system runs on.

#include 
#include 
 
int SymbolsCount;
 
struct IndicatorHandlersStruct
  {
   int SlowMovingAverageHandler;
   int FastMovingAverageHandler;
   int BBoverSTDHandler;
  };
 
struct SymbolData
  {
    string Symbol;
    IndicatorHandlersStruct IndicatorHandlers;
    datetime LastBarTime;
  };
 
SymbolData SymbolsTable[30];
 
VirtualAccount VirtualSystem;    
 
datetime VirtualLastBarTime;
string RealTrading="FORBIDDEN";

The structures are quite similar. The class is instanced in the VirtualSystem object. A VirtualLastBarTime is added to record the open time of the last equity bar from the virtual account. The RealTrading variable will record the trading state with respect to copying trades from the virtual account, that is, “STRAIGHT”, “REVERSED”, or “FORBIDDEN” (only “STRAIGHT” and “FORBIDDEN” are actually used).

The VirtualMakeIndicatorHanders() takes place of the former MakeIndicatorHandlers(). The code is the same, only the name is changed.

void VirtualPositionSetSLTP(int asset_index,int sl,int tp)
  {
   VirtualSystem.VirtualPositionSetSL(asset_index,sl);
   VirtualSystem.VirtualPositionSetSL(asset_index,tp);
  }  
 
void VirtualManagePosition(int asset_index,int operation)
  {
   MqlTradeRequest request;
   MqlTradeResult result;
   int p;
   p=VirtualSystem.VirtualGetPositionType(asset_index);
   double now_volume;
   double current_volume,v0;
   current_volume=VirtualSystem.VirtualGetPositionSize(asset_index);
   v0=current_volume;   
   request.action=TRADE_ACTION_DEAL;
   request.symbol=SymbolsTable[asset_index].Symbol; 
   request.deviation=Slippage;
   request.type_filling=ORDER_FILLING_AON;
   request.type_time=ORDER_TIME_GTC;
   if (p==FLAT)
     {
      request.volume=UnitsToLots( (MarginUsagePerPosition/100)*VirtualSystem.VirtualLastAccountEquity*AccountInfoInteger(ACCOUNT_LEVERAGE),SymbolsTable[asset_index].Symbol );
      if (operation==LONG)
        {
         request.type=ORDER_TYPE_BUY;
         request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_ASK),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));
        }
      if (operation==SHORT)
        {
         request.type=ORDER_TYPE_SELL;
         request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_BID),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));
        }
     }
   else//if (p==FLAT)
     {      
      if (p==LONG&&operation==SHORT)
        {
         request.volume=current_volume+UnitsToLots( (MarginUsagePerPosition/100)*VirtualSystem.VirtualLastAccountEquity*AccountInfoInteger(ACCOUNT_LEVERAGE),SymbolsTable[asset_index].Symbol );
         request.type=ORDER_TYPE_SELL;         
         request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_BID),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));         
        }
      if (p==SHORT&&operation==LONG)
        {
         request.volume=current_volume+UnitsToLots( (MarginUsagePerPosition/100)*VirtualSystem.VirtualLastAccountEquity*AccountInfoInteger(ACCOUNT_LEVERAGE),SymbolsTable[asset_index].Symbol );
         request.type=ORDER_TYPE_BUY;
         request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_ASK),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));
        }
      if (p==LONG&&operation==FLAT)
        {
         request.volume=current_volume;
         request.type=ORDER_TYPE_SELL;         
         request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_BID),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));         
        }
      if (p==SHORT&&operation==FLAT)
        {
         request.volume=current_volume;
         request.type=ORDER_TYPE_BUY;         
         request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_ASK),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));         
        } 
      if (p==LONG&&operation==LONG)//adjustment of the present LONG position
        {
         request.volume=UnitsToLots( (MarginUsagePerPosition/100)*VirtualSystem.VirtualLastAccountEquity*AccountInfoInteger(ACCOUNT_LEVERAGE),SymbolsTable[asset_index].Symbol )-current_volume;
         if (NormalizeDouble(request.volume,2)>0.00)
           {         
            request.type=ORDER_TYPE_BUY;
            request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_ASK),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));
           }
         else //we close the difference
           {
            request.volume=MathAbs(request.volume);
            request.type=ORDER_TYPE_SELL;         
            request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_BID),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));         
           }
        }        
      if (p==SHORT&&operation==SHORT)//adjustment of the present SHORT position
        {
         request.volume=UnitsToLots( (MarginUsagePerPosition/100)*VirtualSystem.VirtualLastAccountEquity*AccountInfoInteger(ACCOUNT_LEVERAGE),SymbolsTable[asset_index].Symbol )-current_volume;
         if (NormalizeDouble(request.volume,2)>0.00)
           {
            request.type=ORDER_TYPE_SELL;         
            request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_BID),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));                  
           }
         else
           {
            request.volume=MathAbs(request.volume);
            request.type=ORDER_TYPE_BUY;
            request.price=NormalizeDouble(SymbolInfoDouble(SymbolsTable[asset_index].Symbol,SYMBOL_ASK),SymbolInfoInteger(SymbolsTable[asset_index].Symbol,SYMBOL_DIGITS));            
           }
        }        
     }//if (p==FLAT)
   if (request.volume>0.00)
     {       
      VirtualSystem.VirtualChangePosition(asset_index,request.type,request.price,request.volume);
      VirtualPositionSetSLTP(asset_index,VirtualStopLoss,VirtualTakeProfit);      
     }
   //deciding to replicate , stay out or reverse signals on real portfolio
   if (RealTrading!="FORBIDDEN")
     {
      if (RealTrading=="STRAIGHT")
        ManagePosition(asset_index,operation);
      else
        {
         if (operation==LONG)
           ManagePosition(asset_index,SHORT);
         if (operation==SHORT)
           ManagePosition(asset_index,LONG);
         if (operation==FLAT)
           ManagePosition(asset_index,FLAT);
        }
     }
   return;     
  }

This is the VirtualManagePosition(). It operates like ManagePosition() however its requests are directed towards the virtual account. At the end, after analysing the RealTrading variable, it decides whether to do nothing, to copy or to reverse virtual operations on the actual trading account.

The MASignal(), PositionSetSLTP(), and ManagePosition() remain the same as in the original expert.

void VirtualTradeLogics(int asset_index)
  {
   string s;
   double stddev[1];
   double bbhi[1];
   double bblo[1];
   int copycount;
   s=MASignal(asset_index);    
   if (VolatilityFilter==true)
     {
      copycount=CopyBuffer(SymbolsTable[asset_index].IndicatorHandlers.BBoverSTDHandler,0,0,1,stddev);
      copycount=CopyBuffer(SymbolsTable[asset_index].IndicatorHandlers.BBoverSTDHandler,2,0,1,bbhi);
     }
   if (VirtualSystem.VirtualGetPositionType(asset_index)==FLAT)
     {      
      if (s=="L")
        {
         if (VolatilityFilter==true)
           {
            if (stddev[0]>bbhi[0])
              VirtualManagePosition(asset_index,LONG);
           }
         else
           VirtualManagePosition(asset_index,LONG);
        }
      if (s=="S")
        {
         if (VolatilityFilter==true)
           {
            if (stddev[0]>bbhi[0])
              VirtualManagePosition(asset_index,SHORT);
           }
         else
           VirtualManagePosition(asset_index,SHORT);
        }
     }//else if (GetPositionType(asset_index)==FLAT)
   else//position is not flat
     {
      if (VirtualSystem.VirtualGetPositionType(asset_index)==LONG)
        {        
         if (VolatilityFilter==true)
           {
            if (s=="S"||stddev[0]<bbhi[0])
              VirtualManagePosition(asset_index,FLAT);
           }           
         else
           {            
            if (s=="S")
              {               
               VirtualManagePosition(asset_index,FLAT);           
              }
           }//if (VolatilityFilter==true)
        }
      if (VirtualSystem.VirtualGetPositionType(asset_index)==SHORT)
        {         
         if (VolatilityFilter==true)
           {
            if (s=="L"||stddev[0]<bbhi[0])
              VirtualManagePosition(asset_index,FLAT);
           }
         else
           {            
            if (s=="L")
              {               
               VirtualManagePosition(asset_index,FLAT);           
              }
           }//if (VolatilityFilter==true)        
        }//if (VirtualSystem.VirtualGetPositionType(asset_index)==SHORT)
     }           
   return;
  }

This is the former TradeLogics() , but modified in order to operate on the virtual account.

int PeriodToMinutes(ENUM_TIMEFRAMES p)
  {
   switch(p)
     {
      case PERIOD_M1:return(1);
      case PERIOD_M2:return(2);
      case PERIOD_M3:return(3);
      case PERIOD_M4:return(4);
      case PERIOD_M5:return(5);
      case PERIOD_M6:return(6);
      case PERIOD_M10:return(10);
      case PERIOD_M12:return(12);
      case PERIOD_M15:return(15);
      case PERIOD_M20:return(20);
      case PERIOD_M30:return(30);
      case PERIOD_H1:return(60);
      case PERIOD_H2:return(60*2);
      case PERIOD_H3:return(60*3);
      case PERIOD_H4:return(60*4);
      case PERIOD_H6:return(60*6);
      case PERIOD_H8:return(60*8);
      case PERIOD_H12:return(60*12);
      case PERIOD_D1:return(60*24);
      case PERIOD_W1:return(60*24*7);
      case PERIOD_MN1:return(60*24*30);
      default: return(0);
     }
  };

This function is used in OnInit() to setup the virtual account’s bar timespan , and it converts periods into minutes.

//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+
int OnInit()
  {
   SymbolsTable[0].Symbol="EURUSD";
   SymbolsTable[1].Symbol="USDJPY";
   SymbolsTable[2].Symbol="GBPUSD";
   SymbolsTable[3].Symbol="USDCHF";
   SymbolsCount=1;
 
   VirtualSystem.VirtualAccountSet(AccountInfoDouble(ACCOUNT_EQUITY),PeriodToMinutes(Period())*60);  
   VirtualSystem.VirtualSymbolsCount=SymbolsCount;      
   for (int i=0;i<SymbolsCount;i++)
      VirtualSystem.VirtualSymbolsTable[i].Symbol=SymbolsTable[i].Symbol;
   VirtualMakeIndicatorHandlers();
   VirtualLastBarTime=VirtualSystem.VirtualEquity[VirtualSystem.VirtualLastEquityPos].opentime;
   RealTrading="FORBIDDEN";
   return(0);
  }

This is OnInit(). It prepares symbols like former OnInit() then copies symbols to the virtual account. Sets up virtual account and makes sure RealTrading is set to “FORBIDDEN”.

//+------------------------------------------------------------------+
//| Expert tick function                                             |
//+------------------------------------------------------------------+
void OnTick()
  {   
   double loweq;         
   double fast_ma_now,slow_ma_now,fast_ma_old,slow_ma_old;
   double EquityCloseBuffer[31];
   datetime datetime_array[1];   
   //marking to market virtual positions
   VirtualSystem.VirtualMarkToMarket(-1);
   //looping symbols to apply logics
   for (int i=0;i<SymbolsCount;i++)
      {
       CopyTime(SymbolsTable[i].Symbol,Period(),0,1,datetime_array);
       if (datetime_array[0]!=SymbolsTable[i].LastBarTime)
         {                  
          VirtualTradeLogics(i);
          SymbolsTable[i].LastBarTime=datetime_array[0];
         }//if (datetime_array[0]!=LastBarOccurred[i])
      }//for (int i=0;i
 
   //now we check if we have enough equity bars ; if true, we analyse to see whether we activate or not live trading
   if (VirtualSystem.VirtualEquity[VirtualSystem.VirtualLastEquityPos].opentime!=VirtualLastBarTime)
     {
      if (VirtualSystem.VirtualLastEquityPos>ArrayRange(EquityCloseBuffer,0))  
        {
         for (int i=1;i<=ArrayRange(EquityCloseBuffer,0)-1;i++)
            {         
             EquityCloseBuffer[i]=StringToDouble( DoubleToString(VirtualSystem.VirtualEquity[VirtualSystem.VirtualLastEquityPos-ArrayRange(EquityCloseBuffer,0)+1+i].close,2) );             
             loweq=StringToDouble( DoubleToString(VirtualSystem.VirtualEquity[VirtualSystem.VirtualLastEquityPos-ArrayRange(EquityCloseBuffer,0)+1+i].low,2));                
            }                                    
        }
      fast_ma_now=SimpleMA(ArrayRange(EquityCloseBuffer,0)-1,EquityFastMAPeriod,EquityCloseBuffer);
      slow_ma_now=SimpleMA(ArrayRange(EquityCloseBuffer,0)-1,EquitySlowMAPeriod,EquityCloseBuffer);
      fast_ma_old=SimpleMA(ArrayRange(EquityCloseBuffer,0)-2,EquityFastMAPeriod,EquityCloseBuffer);
      slow_ma_old=SimpleMA(ArrayRange(EquityCloseBuffer,0)-2,EquitySlowMAPeriod,EquityCloseBuffer);
      //bullish ma cross on virtual system's equity line      
      if (fast_ma_now>slow_ma_now&&fast_ma_old<slow_ma_old)
        {          
         RealTrading="STRAIGHT";
         //replicating virtual portfolio into real
         //position sizes are calculated to be proportional with the real account equity
         for (int i=0;i<SymbolsCount;i++)
            ManagePosition(i,VirtualSystem.VirtualGetPositionType(i), 
                    UnitsToLots(
                     (VirtualSystem.VirtualGetPositionSize(i)*
                      AccountInfoDouble(ACCOUNT_EQUITY)/
                      VirtualSystem.VirtualLastAccountEquity)*
                      SymbolInfoDouble(SymbolsTable[i].Symbol,SYMBOL_TRADE_CONTRACT_SIZE)
                      ,SymbolsTable[i].Symbol) );
        }
      //bearish ma cross on virtual system's equity line        
      if (fast_ma_now<slow_ma_now&&fast_ma_old>slow_ma_old)
        {
         RealTrading="FORBIDDEN";
         //closing real positions
         for (int i=0;i<SymbolsCount;i++)
            ManagePosition(i,FLAT);         
        }             
      VirtualLastBarTime=VirtualSystem.VirtualEquity[VirtualSystem.VirtualLastEquityPos].opentime;
     }
   return;
  }

This is the OnTick(). Like former OnTick(), it has a symbol loop and applies the logics for each symbol, on the virtual account this time. It declares the EquityCloseBuffer[] where last values of virtual acount equity bars are copied, after each loop is finished. When this buffer contains enough values, a moving average cross on the virtual account equity line is calculated. When these moving average yield a bullish signal, RealTrading is set to “STRAIGHT”. Portfolio is being replicated from the virtual account on the actual account, proportional to the size of actual account. Of course, the VirtualManagePosition() will have subsequent calls to ManagePosition() and from this time on both accounts will work in the same time. When the moving average cross becomes bearish, RealTrading is set to “FORBIDDEN” and actual account positions are closed.

The OnDeinit() is exactly the same, as the indicator handlers are residing in the same SymbolsTable[].

Surprised? The portfolio run is more unstable than one pair run, as it was for the main trading system. The extra volatility translated into range movements on the equity line, which in turn translated into false positive signals given by moving averages on the equity line, making the virtualization to be a fiasco. There are at least three points to consider:

1. Sampling frequency
What is the optimal sampling frequency for the equity moving average? After all, we don’t know exactly how MetaTrader compresses equity data on bars so we don’t know its own sampling interval. Then, how to sample for the average ?

2. Optimal trading system for the equity line
Nobody said that moving averages over equity are the best way to fit. Other criterias may apply, same as in regular trading.

3. Should you trade all the instruments in the same virtual account, or trade in separate virtual accounts and fit each one’s equity ?

The main conclusion that yields is that trading systems are not to be based on cold, pure application of indicators. Indicators, as their name says, are indicating some of the actions that have to be taken, as counselors that give advices, influencing positively the results of trading systems, but their place is not at the core of trading systems. Rather, sound statistics and arbitrage relationships are at the base of effective trading systems.

File links:
TradeVirtualization.mqh
VirtualPortfolioMA.mq5

MQL5 : Implementation of a simple CEP engine

Bogdan Caramalac, MQLmagazine sr.editor — Mon, 24 May 2010 21:28:51 +0000

[Versiunea romaneasca] [MQLmagazine.com in romana] [English edition]

This edition was harder to make than any of the previous editions. The late release of Strategy Tester and the large number of bugs that surged while making the CEP engine took a lot of time. From this moment, magazine issues will not be monthly anymore, because they will contain a lot of work and updates of past works. This is why, this time we are adding the Forum to the Magazine.

Now back to the CEP engine.

The CEP engine was some of the hardest stuff possible to write in MQL5. It was hard not only for me as programmer, but it had implications over MT5 as well, because the spaghetti data structure that it uses put MetaTrader to the test ; it needed a lot of bug filtering, especially on objects and memory allocations. The code managed to run with minor issues barely on build 270 (yes, already in May), and completely clean on build 271. It’s a long hard code, and I’ll chop it to pieces for commenting. I also left inline comments and Print lines intact to be better understood by readers. The algorithm is quite modified, not the same as the one presented in Anatomy of a Simple CEP Engine .

The following file it’s gonna be called CEPEngine.mqh :

//+------------------------------------------------------------------+
//|                                                    CEPEngine.mqh |
//|                                       Copyright Bogdan Caramalac |
//|                                           http://mqlmagazine.com |
//+------------------------------------------------------------------+
#property copyright "Bogdan Caramalac"
#property link      "http://mqlmagazine.com"
 
#property version "1.0";
 
//*******************************************
 
string Replicate(string s,int n)
   {
   string res="";
   int k=1;
   if (n<1)
      return(res);                    
   for (k=1;k<=n;k++) 
       {
        StringConcatenate(res,res,s);
       }
    return(res);
   }          
 
string BoolToString(bool b)
  {
   if (b==true)
     return(".T.");
   else
     return(".F.");   
  }
 
//*******************************************

These are very simple string functions, that are needed for the larger functions that translate structures into string messages, descriptions of several CEP structures. The first function will replicate a string, and the second one will return a string for a boolean , in a FoxPro-ish style.

struct SimpleEvent
  {
   public:
   uint EventID;
   uint EventTime;
   uint Reserved;
   ushort BindCount;
   uint Binds[100];
  };
 
   bool BindEvent(uint eventid,SimpleEvent& s)
     {
      bool fd;
      if (s.BindCount==0)
        {
         s.Binds[0]=eventid;
         s.BindCount++;
         return(true);
        }
      else
        {
         fd=false;
         //checking previous bind
         for (int i=0;i<s.BindCount;i++)
            {
             if (s.Binds[i]==eventid)
               {
                fd=true;
                break;
               }
            }//for (int i=0;i
         if (fd==false)
           {
            if (s.BindCount<ArrayRange(s.Binds,0)-1)
              {
               s.Binds[s.BindCount]=eventid;
               s.BindCount++;
               return(true);
              }
            else
              return(false);
           }
         else
           return(false);
        }//else if (s.BindCount!=0)
     }
 
   int FindBind(uint eventid,SimpleEvent& s)
     {
      bool fd;
      if (s.BindCount==0)
        {
         return(-1);
        }
      else
        {
         fd=false;
         //checking previous bind
         for (int i=0;i<s.BindCount;i++)
            {
             if (s.Binds[i]==eventid)
               {
                return(i);
                break;
               }
            }//for (int i=0;i
         return(-1);
        }
     }     
 
   bool UnbindEvent(uint eventid,SimpleEvent& s)
     {
      if (s.BindCount==0)
        return(false);
      int fd;
      fd=-1;
      for (int i=0;i<s.BindCount;i++)
         {
          if (s.Binds[i]==eventid)
            {
             fd=i;
             break;
            }
         }//for (int i=0;i
      if (fd==0)
        return(false);
      else
        {
         for (int j=fd+1;j<s.BindCount;j++)
            {
             s.Binds[j-1]=s.Binds[j];
            }
         s.Binds[s.BindCount-1]=0;
         s.BindCount--;
         return(true);
        }//if (fd==0)
     }

The SimpleEvent structure is at the core of the engine. The events queue that I described in my previous article it is made by tiny SimpleEvent structures. EventID plays the ID, EventTime is the time (in ticks – milliseconds since Windows start – not datetime format), BindCount tells to how many complex events the simple event is binded, and the Binds[] contains these events. The BindEvent(), FindBind() and UnbindEvent() functions are to work with this structure. Normally, I could have opted for a class instead of a structure, but what you gain in class functionality you lose in data maneuvrability : you cannot copy objects, but field by field, otherwise you get Structure have objects and cannot be copied.

struct SimpleEventArrangement
  {
   int EventCount;
   uint EventArray[100];
   int Prebinds[100];
   int NegationsCount[100]; //how many negation events before each event from the array
   uint NegationsArray[10][100]; //store negation events before each event from the array
   bool NegationsChecked[100];//checks of the first negations that match
  };
 
struct ComplexEvent
  {
   uint ComplexID;
   uint TimeConstraint;
   ushort ArrangementsCount;
   SimpleEventArrangement ArrangementsArray[100];
  };

The ComplexEvent structure is the base unit of the complex events table. Remember the pic from the previous article? That is quite what can be found in the ComplexEvent structure. Thus, we have the ID , time constraint, arrangements count (number of rows) , or versions of event queues that make up the same complex event, and finally an arrangements array, made up by SimpleEventArrangement(s).

The SimpleEventArrangement structure describes what can be called to be a row of that matrix. Thus we have a count, and the row (meaningly, the EventArray[]) , which contains the required simple events. The Prebinds[] will note down of course, the prebinds of these events, meaningly the position in the event queue for each simple event that is found. The NegationsCount[] will contain how many negation events are required not to appear before every simple event that is required. Thus, one count per each required simple event. The NegationsArray[] will contain these events required not to appear, on columns, rather than on lines. The NegationsChecked[] array is a sort of shortened prebind array. Instead of storing where the negations appear, a simple negation flag is raised in this array, thus the array is unidimensional.

void MakeSimpleEventArrangement(string s,SimpleEventArrangement& arrg)
  {
   int len=StringLen(s+" ");
   string rdnow="";
   string c;
   uint number,swap;
   int normcount;
   arrg.EventCount=0;
   string first;
   int intg,d,crt;
   double dbl;
   s=s+" ";//to make sure it registers last word
   ArrayInitialize(arrg.EventArray,0);
   ArrayInitialize(arrg.NegationsCount,0);
   ArrayInitialize(arrg.NegationsArray,0);
   ArrayInitialize(arrg.Prebinds,-1);
   ArrayInitialize(arrg.NegationsChecked,false);
   normcount=-1;
   if (len!=0)
     {
      for (int i=0;i<len;i++)
         {
          c=StringSubstr(s,i,1);
          if (c==" ")
            {
             if (rdnow!="")
               {
                first=StringSubstr(rdnow,0,1);
                if (first=="-"||first=="!"||first=="n"||first=="N")
                  {
                   number=StringToInteger(StringSubstr(rdnow,1,StringLen(rdnow)-1));
                   if (normcount!=-1)   
                     {
                      arrg.NegationsArray[normcount][arrg.NegationsCount[normcount]]=number;
                      arrg.NegationsCount[normcount]++;
                     }                  
                  }
                else
                  {
                   normcount++;
                   number=StringToInteger(StringSubstr(rdnow,0,StringLen(rdnow) ));
                   arrg.EventArray[normcount]=number;
                   arrg.EventCount=normcount+1;
                  }
                rdnow="";
               }//if (rdnow!="")                
            }//if (c==" ")
          else
            {
             StringConcatenate(rdnow,rdnow,c);
            }
         }//for (int i=0;i
       if (rdnow!="")
         {
          first=StringSubstr(rdnow,0,1);
          if (first=="-"||first=="!"||first=="n"||first=="N")
            {
             number=StringToInteger(StringSubstr(rdnow,1,StringLen(rdnow)-1));
             if (normcount!=-1)   
               {
                arrg.NegationsArray[normcount][arrg.NegationsCount[normcount]]=number;
                arrg.NegationsCount[normcount]++;
               }                  
            }
          else
            {
             normcount++;
             number=StringToInteger(StringSubstr(rdnow,0,StringLen(rdnow) ));
             arrg.EventArray[normcount]=number;
             arrg.EventCount=normcount+1;
            }
          rdnow="";
         }//if (rdnow!="")
 
      //reversing event order
      if (arrg.EventCount>0)
        {
         dbl=NormalizeDouble(arrg.EventCount/2,0);
         intg=NormalizeDouble(MathRound(NormalizeDouble(arrg.EventCount,0)/2),0);
         if (dbl==intg)
           d=0;
         else
           d=1;           
         crt=0;         
         for (int i=0;i<=intg-1-d;i++)
            {             
             swap=arrg.EventArray[i];
             arrg.EventArray[i]=arrg.EventArray[arrg.EventCount-1-crt];
             arrg.EventArray[arrg.EventCount-1-crt]=swap;             
             for (int l=0;l<ArrayRange(arrg.NegationsArray,0);l++)
                {                 
                 swap=arrg.NegationsArray[l][i];
                 arrg.NegationsArray[l][i]=arrg.NegationsArray[l][arrg.EventCount-1-crt];
                 arrg.NegationsArray[l][arrg.EventCount-1-crt]=swap;
                }
             crt++;
            }
        }
 
     }//if (len!=0)
   return;
  }

The MakeSimpleArrangement() function will build a simple arrangement, from a string, as instructed by the user. For instance, “10 !6 11″ means “10, followed by 11, with a 6 that has not to appear before 11″. Spacebar is separator, and negation flag can be “-”,”!”,”n” or “N”. Note that at the end of the function, data is reversed, from last to first, as I said in the previous article, that complex event arrangements will start from the nearby events going to the older events. This function has not to be called by the user, rather it is called by MakeComplexEvent().

void MakeComplexEvent(string s,uint eventid,uint time,ComplexEvent &cplx)
  {
   int len=StringLen(s+" ");
   string rdnow="";
   string c;
   string arrg;
   ushort cnt;
   s=s+" ";
   cplx.ComplexID=eventid;
   cplx.TimeConstraint=time;
   cplx.ArrangementsCount=0;
   if (len!=0)
     {
      for (int i=0;i<len;i++)
         {
          c=StringSubstr(s,i,1);
          if (c==";")
            {
             if (rdnow!="")
               {
                MakeSimpleEventArrangement(rdnow,cplx.ArrangementsArray[cplx.ArrangementsCount]);
                cplx.ArrangementsCount++;
                rdnow="";
               }//if (rdnow!="")
            }//if (c==" ")
          else
            StringConcatenate(rdnow,rdnow,c);
         }//for (int i=0;i
      //for last word
      if (rdnow!="")
        {
         MakeSimpleEventArrangement(rdnow,cplx.ArrangementsArray[cplx.ArrangementsCount]);
         cplx.ArrangementsCount++;
         rdnow="";
        }//if (rdnow!="")
     }//if (len!=0)
   return;
  }

The MakeComplexEvent() function is to be called by user when telling the engine how the complex events look like. The definition is a big string with more simple event arrangements, separated by semicolon. The other parameters are the complex event ID, the time constraint and a complex event variable to store the result in.

string ComplexEventToString(ComplexEvent& c)
  {
   string res="";
   res=res+"ComplexID "+DoubleToString(c.ComplexID,0)+"\n";
   res=res+"Arrangements "+DoubleToString(c.ArrangementsCount,0)+"\n";
   for (int i=0;i<c.ArrangementsCount;i++)
      {
       res=res+"Events: "+DoubleToString(c.ArrangementsArray[i].EventCount,0)+"\n";
       for (int j=0;j<c.ArrangementsArray[i].EventCount;j++)
          {
           res=res+" "+DoubleToString(c.ArrangementsArray[i].EventArray[j],0);
          }
       res=res+"\n";       
       res=res+"Negations:\n";
       for (int j=0;j<10;j++)
          {
           for (int k=0;k<c.ArrangementsArray[i].EventCount;k++)
              {
               res=res+" "+DoubleToString(c.ArrangementsArray[i].NegationsArray[j][k],0);
              }
           res=res+"\n";
          }
       res=res+"\n";
      }
   return(res);
  }

This function was made for commenting reasons only. The function describes a complex event in a string form. You can output the resulting string with Comment().

class CEPEngine
  {
   //setup elements
   private:
   uint QueueTimeConstraint;
   ComplexEvent ComplexEventsTable[30];
   ushort ComplexEventsCount;
   //data
   SimpleEvent EventsQueue[1000];    
   int OldestIndex;
   uint OldestTime;
 
   public:
   void CEPEngine();//constructor
   void ClearEvents();//cleans event table
   void AddComplexEvent(ComplexEvent &c);   
   virtual void EventsCallback(uint EventID);//here you treat raised events from the CEP Engine
   void SetupEngineMilliseconds(uint ms);
   void RaiseEvent(uint EventID);
   void EngineRun();
   string StatusString();
   };

Finally, we got to the CEP engine class.
QueueTimeConstraint is an important variable, which tells which is the length in time for which the engine will look in the queue. Incoming simple events, that are found in the EventsQueue[] , and go past this “event horizon” are overwritten. The OldestIndex is the end of the array ; it is a dynamic end, calculated at every new RaiseEvent() call from the outside, abiding the QueueTimeConstraint, and it has the time , in ticks, in OldestTime.

Now the small stuff:

   void CEPEngine::ClearEvents()  //cleans event table
     {
      for (int i=0;i<ArrayRange(ComplexEventsTable,0);i++)
         {
          ComplexEventsTable[i].ArrangementsCount=0;
         }
      for (int i=0;i<ArrayRange(EventsQueue,0);i++)
          {
           EventsQueue[i].EventID=0;
           EventsQueue[i].EventTime=0;
           EventsQueue[i].BindCount=0;
           ArrayInitialize(EventsQueue[i].Binds,0);
          }
      OldestIndex=-1;
      OldestTime=0;
      ComplexEventsCount=0;
     }  
 
   void CEPEngine::CEPEngine()
     {
      ClearEvents();
     }
 
   void CEPEngine::AddComplexEvent(ComplexEvent &c)
     {
      ComplexEventsTable[ComplexEventsCount]=c;
      ComplexEventsCount=ComplexEventsCount+1;
     }
 
 
   void CEPEngine::SetupEngineMilliseconds(uint ms)
     {
      QueueTimeConstraint=ms;
      return;
     }
 
 
   void CEPEngine::EventsCallback(uint EventID)//here you treat raised events from the CEP Engine
     {
     }   
 
    string CEPEngine::StatusString()
       {
        string res="";
        string eventtext;
        int etl;
        if (ComplexEventsCount!=0)
          {
           for (int c=0;c<ComplexEventsCount;c++)
              {
               eventtext="["+DoubleToString(ComplexEventsTable[c].ComplexID,0)+"]::";
               etl=StringLen(eventtext);
               res=res+eventtext;
               if (ComplexEventsTable[c].ArrangementsCount!=0)
                 {
                  for (int a=0;a<ComplexEventsTable[c].ArrangementsCount;a++)
                     {                     
                      if (ComplexEventsTable[c].ArrangementsArray[a].EventCount!=0)
                        {                         
                         for (int e=0;e<ComplexEventsTable[c].ArrangementsArray[a].EventCount;e++)
                            {
                             res=res+DoubleToString(ComplexEventsTable[c].ArrangementsArray[a].Prebinds[e],0)+"  ";
                            }
                         res=res+"\n"+Replicate(" ",etl);
                         for (int e=0;e<ComplexEventsTable[c].ArrangementsArray[a].EventCount;e++)
                            {
                             res=res+BoolToString(ComplexEventsTable[c].ArrangementsArray[a].NegationsChecked[e])+"  ";
                            }
                         res=res+"\n";
                         if (a!=ComplexEventsTable[c].ArrangementsCount-1)
                           {
                            res=res+Replicate(" ",etl);
                           }                      
                        }//if (ComplexEventsTable[c].ArrangementsArray[a].EventCount!=0)                   
                     }//for (int a=0;a
                 }//if (ComplexEventsTable[c].ArrangementsCount!=0)
              }//for (int c=0;c
          }//if (ComplexEventsCount!=0) 
        return(res);
       }

So, the ClearEvents() clears the simple events queue. This is the reason for which it is called from the CEPEngine() constructor. The AddComplexEvent() adds a complex event to the table. It has to be called by user, after making complex events with MakeComplexEvent(). The engine is ready to work after its time constraint is set up by the user with SetupEngineMilliseconds(). And finally, the EventsCallback() is the virtual method that has to be overridden by the user, being called when the CEP engine raises a complex event.
The StatusString() function produces a comment version of the engine status.

Now some average complicated stuff: RaiseEvent() : when the user raises a simple event

    void CEPEngine::RaiseEvent(uint EventID)
      {
       long time=GetTickCount();
       bool isforbidden=true;
       uint opposite=0;
       uint difft;
       if (OldestIndex>0)
         {
          for (int i=0;i<ArrayRange(EventsQueue,0);i++)
             {
              if (EventsQueue[i].EventTime==0)
                {
                 OldestIndex=i-1;
                 OldestTime=EventsQueue[i-1].EventTime;
                 break;
                }
              if (time-EventsQueue[i].EventTime>QueueTimeConstraint)
                {
                 OldestIndex=i-1;
                 OldestTime=EventsQueue[i-1].EventTime;
                 break;
                }//if (time-EventsQueue[i].EventTime>QueueTimeConstraint)  
             }//for (int i=0;i
         }//if (OldestIndex!=0)
       else
         {
          if (OldestIndex<=0)
            {
             OldestIndex=0;
             OldestTime=time;
            }
         }
       for (int j=OldestIndex+1;j>0;j--)
          {
           EventsQueue[j].EventID=EventsQueue[j-1].EventID;
           EventsQueue[j].EventTime=EventsQueue[j-1].EventTime;
           EventsQueue[j].BindCount=EventsQueue[j-1].BindCount;
          }
       ArrayInitialize(EventsQueue[0].Binds,0);
       EventsQueue[0].EventID=EventID;
       EventsQueue[0].EventTime=time;
       EventsQueue[0].BindCount=0;
       OldestIndex=OldestIndex+1;
 
       if (ComplexEventsCount!=0)
         {
          EngineRun();
         }         
       return;
      }//raise event ends

The function has a moderate degree of complexity.
Lines 7 – 32 establish the event horizon within the queue. That is, which is the oldest event within the queue, abiding the time constraint of the whole queue.
Lines 33 – 38 move the queue one step lower , making room for a new simple event.
Lines 39 – 43 insert the new simple event coming from the user program.
Lines 45 – 48 call the EngineRun() method is to process the new data, if at least one complex event is registered in the table.

At the heart of the CEP engine lies the EngineRun() method. It is the most complicated piece of code of the entire class. But, there would be no car without the engine, right?

   void CEPEngine::EngineRun()
     {
      long time=GetTickCount();
      bool any_binded;
      int CrtColumn[100][100];//address CrtColumn as CrtColumn(event,arrangement)
      bool bypass=false;
      int q,p;
      int j,lastj;
      int test;
      if (ComplexEventsCount==0)
        return;
      ArrayInitialize(CrtColumn,0);
      for (int c=0;c<ComplexEventsCount;c++)
         {
          Print("Arrangements in complex event ",c," are ",ComplexEventsTable[c].ArrangementsCount);
          if (ComplexEventsTable[c].ArrangementsCount!=0)
            {                          
             for (int a=0;a<ComplexEventsTable[c].ArrangementsCount;a++)
                {      
                 //Check if the current simple event within arrangement within current complex event is to be binded
                 //by checking with simple events from the queue
                 p=0;                                       
                 while (time-EventsQueue[p].EventTime<=ComplexEventsTable[c].TimeConstraint)
                   {
                    Print("Looking on entry ",p," where is simple event ",EventsQueue[p].EventID," comparing with ",ComplexEventsTable[c].ArrangementsArray[a].EventArray[CrtColumn[c][a]]," on complex event ",c,", arrg. ",a," (column ",CrtColumn[c][a],") vs count ",ComplexEventsTable[c].ArrangementsArray[a].EventCount);                       
                    if (EventsQueue[p].EventID==ComplexEventsTable[c].ArrangementsArray[a].EventArray[CrtColumn[c][a]])
                      {
                       Print("Found simple event ",EventsQueue[p].EventID," for column ",CrtColumn[c][a]," of arrangement ",a," of event ",c," meaningly id=",ComplexEventsTable[c].ComplexID); 
                       ComplexEventsTable[c].ArrangementsArray[a].Prebinds[CrtColumn[c][a]]=p;
                       CrtColumn[c][a]++;                                                                                        
                       bypass=false;
                       if (CrtColumn[c][a]>1)
                         {
                          q=ComplexEventsTable[c].ArrangementsArray[a].Prebinds[CrtColumn[c][a]-2];
                          //-2 instead of -1 because we already incremented the CrtColumn by ++
                          Print("Checking negations from previous bind, ",q," to current element ",p);
                          q=q+1;
                          //Checking negations; if test passes, simple event is prebinded;                                                          
                          //q is an older event than the current one, p, so q>p
                          if (ComplexEventsTable[c].ArrangementsArray[a].NegationsCount[a]!=0)
                            {                                                
                             while (q<p)
                               {
                                for (int j=0;j<ComplexEventsTable[c].ArrangementsArray[a].NegationsCount[a];j++)
                                   {
                                    lastj=j;
                                    Print("Comparing event on ",q," meaningly ",EventsQueue[q].EventID," with negation ",ComplexEventsTable[c].ArrangementsArray[a].NegationsArray[j][CrtColumn[c][a]-2+1]);
                                    if (EventsQueue[q].EventID==ComplexEventsTable[c].ArrangementsArray[a].NegationsArray[j][CrtColumn[c][a]-2+1])
                                      {
                                       ComplexEventsTable[c].ArrangementsArray[a].NegationsChecked[j]=true;
                                       break;
                                      }
                                   }
                                 if (ComplexEventsTable[c].ArrangementsArray[a].NegationsChecked[lastj]==true)
                                   break;
                                 q++;
                                }
                            }//if (ComplexEventsTable[c].ArrangementsArray[a].NegationsCount!=0)
                          if (ComplexEventsTable[c].ArrangementsArray[a].NegationsChecked[lastj]==true)
                            {
                             Print("Erasing last bind on ",CrtColumn[c][a]-1,". Decreasing current column for complex ",c,", arrangement ",a);
                             ComplexEventsTable[c].ArrangementsArray[a].Prebinds[CrtColumn[c][a]-1]=-1;//erase prebind;                                  
                             CrtColumn[c][a]--;
                             bypass=true;
                             break;
                            } 
                         }//if (CrtColumn[c,a]>1)                            
                       if (bypass==false)//meaning,if no negations are checked                          
                         {
                          if (ComplexEventsTable[c].ArrangementsArray[a].EventCount==CrtColumn[c][a])//all prebinds are done
                            {
                             //checking time constraint again
                             if (GetTickCount()-EventsQueue[ComplexEventsTable[c].ArrangementsArray[a].Prebinds[ComplexEventsTable[c].ArrangementsArray[a].EventCount-1]].EventTime<=ComplexEventsTable[c].TimeConstraint)
                               {
                                //before binding, will check if any of these events is previously binded to the same event
                                any_binded=false;
                                for (int e=0;e<ComplexEventsTable[c].ArrangementsArray[a].EventCount;e++)
                                   {
                                    Print("Prebind",ComplexEventsTable[c].ArrangementsArray[a].Prebinds[e]," bind location of complex event id ",ComplexEventsTable[c].ComplexID," : ",FindBind(ComplexEventsTable[c].ComplexID,EventsQueue[ComplexEventsTable[c].ArrangementsArray[a].Prebinds[e]]));
                                    if (FindBind(ComplexEventsTable[c].ComplexID,EventsQueue[ComplexEventsTable[c].ArrangementsArray[a].Prebinds[e]])!=-1)
                                      {
                                       any_binded=true;
                                       Print("One event is already binded to complex event ",ComplexEventsTable[c].ComplexID);
                                       break;
                                      }
                                   }
                                if (any_binded==false)
                                  {
                                   //binding event
                                   for (int e=0;e<ComplexEventsTable[c].ArrangementsArray[a].EventCount;e++)
                                      {
                                       Print("Binding simple event on ",ComplexEventsTable[c].ArrangementsArray[a].Prebinds[e]," to ",ComplexEventsTable[c].ComplexID);
                                       BindEvent(ComplexEventsTable[c].ComplexID,EventsQueue[ComplexEventsTable[c].ArrangementsArray[a].Prebinds[e]]);
                                      }                       
                                   EventsCallback(ComplexEventsTable[c].ComplexID);//raise complex event for user to handle                                   
                                  }//if (any_binded==false)                                
                                break;//search exhausted for current arrangement of current complex event
                                      //it's either complete, so pointless to look forward
                                      //or one of the simple events is binded to the same complex event, again pointless
                               }//if (GetTickCount()-EventsQueue[ComplexEventsTable[c].ArrangementArray[a].Prebinds[...]].EventTime<=ComplexEventsTable[c].TimeConstraint)
                             else//it's a flop: completed too late; deleting binds
                               {                                      
                                for (int f=0;f<ComplexEventsTable[c].ArrangementsArray[a].EventCount;f++)
                                   {
                                    Print("Unbinding simple event from ",ComplexEventsTable[c].ArrangementsArray[a].Prebinds[f]," to ",ComplexEventsTable[c].ComplexID);
                                    UnbindEvent(ComplexEventsTable[c].ComplexID,EventsQueue[ComplexEventsTable[c].ArrangementsArray[a].Prebinds[f]]);
                                    ComplexEventsTable[c].ArrangementsArray[a].Prebinds[f]=-1;
                                    ComplexEventsTable[c].ArrangementsArray[a].NegationsChecked[f]=false;
                                   }//for (int f=0;f
                                 break;//no point to look forward , time`s up already
                               }//else if (GetTickCount()-EventsQueue[ComplexEventsTable[c].ArrangementArray[a].Prebinds[...]].EventTime<=ComplexEventsTable[c].TimeConstraint)
                            }//if (ComplexEventsTable[c].ArrangementsArray[a].EventCount==CrtColumn[c][a])
                         }//else if (bypass==true)                                                                                                 
                      }//if (EventsQueue[p].EventID==ComplexEventsTable[c].ArrangementsArray[a].EventArray[CrtColumn[c,a]])
                    p++;
                   }//while (time-EventsQueue[p].EventTime<=ComplexEventsTable[c].TimeConstraint)
                }//for (int a=0;a
            }//if (ComplexEventsTable[c].ArrangementsCount!=0)
         }//for (int c=0;c
 
      for (int c=0;c<ComplexEventsCount;c++)
         {          
          if (ComplexEventsTable[c].ArrangementsCount!=0)
            {
             for (int a=0;a<ComplexEventsTable[c].ArrangementsCount;a++)
                {                     
                 if (ComplexEventsTable[c].ArrangementsArray[a].EventCount!=0)
                   { 
                    if (CrtColumn[c][a]<ComplexEventsTable[c].ArrangementsArray[a].EventCount)
                      {
                       //removing prebinds
                       for (int f=0;f<ComplexEventsTable[c].ArrangementsArray[a].EventCount;f++)
                          {
                           ComplexEventsTable[c].ArrangementsArray[a].Prebinds[f]=-1;
                           ComplexEventsTable[c].ArrangementsArray[a].NegationsChecked[f]=false;
                          }//for (int f=0;f
                      }//if (CrtColumn[c][a]
                   }//if (ComplexEventsTable[c].ArrangementsArray[a].EventCount!=0)                       
                }//for (int a=0;a
            }//if (ComplexEventsTable[c].ArrangementsCount!=0)
         }//for (int c=0;c
      return;
     }

The function contains two loops. One is the main loop, lines 13 – 117 , and the second loop is 120 – 140. The second loop has a cleanup role to be explained below.

The main loop will go thru the ComplexEventsTable, to check which appear. For each complex event within this table, the arrangements will be checked, and for each arrangement, a series of procedures will be performed.
Inside this loop, another while loop will pass thru the EventsQueue[], (lines 23 – 115) from the first to the last, looking for events that match the time constraint of the current complex event. One of the most important variables inside this method is the CrtColumn[] array. This one stores the current pointer (current column in the unidimensional EventArray[]). Thus, the procedure checks the current required simple event from the table with the one found in the queue (line 26) , prebinds it and increases the CrtColumn[] value for the current complex event arrangement (lines 29 – 31). Since CrtColumn[] is initialized with zeros, if it finds first event, it is increased to 1. Now, if the CrtColumn[] is higher than one (meaning that the second event was found, and preparing to find the third) it is necessary to check if between the previous two events (e.g. first and the second, in this case) there are negation events (the ones that are forbidden from appearing), only if the NegationsCount[] for that specific simple event is nonzero (lines 40 – 59). If one negation event appears, it will break the search – pointless for the current arrangement. It also sets the bypass variable to true, for the following code, however that is just a programming technique to make the listing look better – the following code will be executed only if no negations appear in the meanwhile. Since events array is zero based, the maximum event that can be binded is EventCount-1, and, if CrtColumn[] for the current arrangement is equal to EventCount, then all events were prebinded. The complex event is like already done. What follows happens only if the oldest event still matches the time constraint. It will check if at least one of the pre-binded events of this pass was binded for real to the analysed complex event, already. In this case, will raise the any_binded flag to be true, and break. If the flag is not raised, then the complex event is considered completed in this arrangement, simple events are binded to the complex event , EventsCallback() is called to notify the user and current search is breaked (lines 90-96) . Otherwise, pre-binding is failed – happens too late. Events are unbinded, negations and prebinds are erased, search it’s breaked – no point to look forward if it’s too late already.

Then what comes is a secondary loop, lines 121 – 141. Since events are mapped from the nearest to the oldest, they are reported to the user creating incomplete complex events whose “shadowing” that manifests into future prohibiting same complex event from being reported, should it appear again, but made by other simple events. All the prebinds are removed, only definitive binds remain valid. Note, however, that the same complex event, conceptually, may be triggered more times than once, by different arrangements, if these contain completely different simple event streaks (e.g. 15 and 16 or 11 and 10 , if all four appear , meaning the same event, both arrangements will be reported as complex events with the same ID, but distinct) because the engine will not be able know if the event appeared again or these mean the same one.

You may question about negations, as you see negations are event specific, each event having its own negation events before it and the previous event. The current engine does not support global negations. However, if you want some events to be global negations, they have to be set up between every two consecutive simple events. The first event in the queue may have negations assigned, however these will not be checked, because there would be no known timeframe where to check for them. Negations are valid only between consecutive simple events.

This is a script example using the CEP engine in a test. The test consist in passing abstract events to the CEP engine. Let’s call it ceptest.mq5 .

//+------------------------------------------------------------------+
//|                                                      ceptest.mq5 |
//|                                        Copyright Bogdan Caramalac|
//|                                           http://mqlmagazine.com |
//+------------------------------------------------------------------+
#property copyright "Copyright 2010, MetaQuotes Software Corp."
#property link      "http://www.mql5.com"
#property version   "1.00"
 
#include 
 
 
//our class: we needed it to override EventsCallback;
class LocalCEPEngine : public CEPEngine
  { 
    void EventsCallback(uint EventID)
     {
      Print("COMPLEX EVENT RECOGNIZED: ",EventID);
     }   
  };
 
//the CEP engine object
LocalCEPEngine myengine;
 
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
  {
   uint tickstart,tick;
   ulong AbstractEvent;
   ComplexEvent c;
   //setting up engine time span
   myengine.SetupEngineMilliseconds(10000);
   //creating complex events
   MakeComplexEvent("10 !6 11 ; 10 12 13 ; 10 13 12",1111,2000,c); //the event for MSFT;  
   myengine.AddComplexEvent(c);   
   //note the events with ! are negations
   MakeComplexEvent("5 !7 15 ; 5 !7 16 17 ; 5 !7 17 !7 !9 16",2222,3000,c); //the event for ^GSPC;
   myengine.AddComplexEvent(c); 
 
   tickstart=GetTickCount();
   tick=tickstart;
 
//alternate tests:
   myengine.RaiseEvent(10);   
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(6);   
   Print("*****************************************************************************************************************");   
   myengine.RaiseEvent(11);   
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(10);
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(11);   
   Print("*****************************************************************************************************************");
   return;
}

This one generates for instance the following output:

I invite you to check the logs, look how the system is working. Try other tests:
Like this:

//alternate tests:
   myengine.RaiseEvent(10);   
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(11);
   Print("*****************************************************************************************************************");
   return;

Or like this:

//alternate tests:
   myengine.RaiseEvent(10);   
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(6);   
   Print("*****************************************************************************************************************");   
   myengine.RaiseEvent(11);   
   Print("*****************************************************************************************************************");

Or like this:

//alternate tests:
   myengine.RaiseEvent(10);   
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(11);   
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(10);
   Print("*****************************************************************************************************************");
   return;

Or like this:

//alternate tests:
   myengine.RaiseEvent(10);   
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(11);   
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(10);
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(11);   
   Print("*****************************************************************************************************************");
   return;

Now you gonna ask me how to find out which combinations of simple events have to be wired up into complex events. Which complex events stay at the basis of the mid frequency algos? How to scout for them?

Well, I didn’t graduate Paul Wilmott’s CQF. I cannot tell you that. I am a simple romanian finance bachelor, possibly going to be fried very soon in the coming mess to the public sector where I sadly work. I gave a minor suggestion about the mechanics of this search, in the article about self-tuning systems. If you got the brains and the power to go forward, and if you believe in the face value of the things, fairness of education and other blah blah stuff – try getting yourself a Computational Finance masters – or a Quantitative Finance masters – should be opening enough. If you can’t finance one, try Finland. I see Hanken is pretty fair – if you have the nerve to pass thru papers and questions. Otherwise, study alone, implement at home – with all the limits coming from this!

File links:
CEPEngine.mqh
ceptest.mq5

Combinatorics in MQL5

Bogdan Caramalac, MQLmagazine sr.editor — Mon, 24 May 2010 21:28:41 +0000

[Versiunea romaneasca] [MQLmagazine.com in romana] [English edition]

Long time ago I wrote on my former blog about generation of permutations, however I never dwelved too much into explaining the algorithm and the code. Indeed, there are situations when combinatorics are to be applied – when scouting for arbitrages or while deploying statistics requiring asset selections as prerequisite.

Permutations

Permutations are quite simple to both find and generate, although math manuals quite skip these algorithms. Since permutations are specific orders in which elements of a set are arranged, that means while arranging the elements, the possibilities to arrange the remainder elements shrink. For instance if we have 4 elements in a set, after setting the first we have just 3 places left, and so on, until the last element fits automatically to the single place left. Thus, the composition of the number depicting the permutation number is done like when composing a number in another numeration base, except that the base shrinks at every step.

For instance we have to find this permutation’s number:

|1|2|3|4|
|1|4|2|3|

At the beginning, the numeration base is 4, and our array is empty:

|1|2|3|4| : mathematical notation of positions
|1|4|2|3| : permutation to decypher

|0|0|0|0| : occupied places in the array
|0|1|2|3| : places in the array (zero-based)
|0|1|2|3| : count array

We look on the first position, 1, and read the first element, “1″. For computing reasons we use zero-based arrays.
Permutation number = 1-1 = 0

|1|2|3|4| : mathematical notation of positions
|1|4|2|3| : permutation to decypher

|0|0|0|0| : presently occupied places in the permutation
|1|0|0|0| : occupied places in positions notation after insertion
|0|1|2|3| : count array (conceptual, for counting)
|x|0|1|2| : new count array for remainder elements

Now we look at the second element. The second element reads “4″. Then, as we have just 3 seats free, the numeration base is 3. Permutation number = (0 x 3) = 0. To this, we add the position in the count array for the element “4″, which is 2 : Permutation number = 0 + 2 = 2

|1|2|3|4| : mathematical notation of positions
|1|4|2|3| : permutation to decypher

|1|0|0|0| : presently occupied places in the notation of positions
|1|0|0|4| : occupied places in positions notation after insertion
|x|0|1|2| : count array
|x|0|1|x| : new count array for remainder elements

Now we look at the third element. The third element reads “2″. Then, as we have just 2 seats free, the numeration base is 2. Permutation number = (2 x 2) = 4. To this, we add the position in the count array for the element “2″, which is 0 : Permutation number = 4 + 0 = 4

|1|2|3|4| : mathematical notation of positions
|1|4|2|3| : permutation to decypher

|1|0|0|4| : presently occupied places in the notation of positions
|1|2|0|4| : occupied places in positions notation after insertion
|x|0|1|2| : count array
|x|x|0|x| : new count array for remainder elements

Calculation stops. We need only to insert in calculation n-1 elements. The nth element is presumed to be on the leftover place.

Final permutation number = 4

The reverse algorithm, to generate permutations from permutation numbers, is the following:
Starting from base 2, we do n-1 steps of dividing the previous result by an increasing base, keeping the remainder.
So we have:
4 / 2 = [2], remainder 0
[2] / 3 = 0, remainder 2
[0] / 4 = 0, remainder 0

Then we start building the permutation back. We read the last remainder, and we find it to be 0. This means our first element is indeed “1″, as 0 from the count array stands for 1 in the positions array.

|1|2|3|4| : mathematical notation of positions
|0|1|2|3| : physical subscripts in array variable
|0|1|2|3| : count array
|1|0|0|0| : occupied positions after current element insertion
|x|0|1|2| : new count array for remainder elements
|1|0|0|0| : built permutation

Our base shrinked to 3, as we have 3 seats left. We read the following remainder, in a reversed order, and we find it to be 2. Looking into the new count array, we see that 2 corresponds to element “4″ from the mathematical notation.

|1|2|3|4| : mathematical notation of positions
|0|1|2|3| : physical subscripts in array variable
|x|0|1|2| : count array
|1|0|0|4| : occupied positions after current element insertion
|x|0|1|x| : new count array for remainder elements
|1|4|0|0| : built permutation

Our base shrinked to 2, as we have 2 seats left. We read the following remainder, in a reversed order, and we find it to be 0. Looking into the new count array, we see that position 0 corresponds to element “2″ from the mathematical notation.

|1|2|3|4| : mathematical notation of positions
|0|1|2|3| : physical subscripts in array variable
|x|0|1|x| : count array
|1|2|0|4| : occupied positions after current element insertion
|x|x|0|x| : new count array for remainder elements
|1|4|0|2| : built permutation

We have only last element to be determined, and that is “3″, on the single free seat unoccupied.

|1|2|3|4| : mathematical notation of positions
|0|1|2|3| : physical subscripts in array variable
|x|x|0|x| : count array
|1|2|3|4| : occupied positions after current element insertion
|x|x|x|x| : new count array for remainder elements
|1|4|3|2| : built permutation

Because both algorithms are quite straight, they can be implemented straight into functions. Thus, we are starting to write the Combinatorics.mqh.

//+------------------------------------------------------------------+
//|                                                Combinatorics.mqh |
//|                                       Copyright Bogdan Caramalac |
//|                                           http://mqlmagazine.com |
//+------------------------------------------------------------------+
#property copyright "Copyright Bogdan Caramalac"
#property link      "http://mqlmagazine.com"
 
//*************************************************
 
 string Replicate(string s, int count)
   {
    string res="";
    if (count<=0)
      return(res);
    for (int i=1;i<=count;i++)
      StringConcatenate(res,res,s);
    return(res);
   }
 
 int MathDiv(int a,int b)
    {
     int res;
     res=a-MathRound(MathMod(a,b));
     res=MathRound(res/b);
     return(res);
    }
 
 int Round(double a)
    {
     return(MathRound(NormalizeDouble(a,0)));
    }
 
//*************************************************

Usual stuff that we don’t insist on, we go forward to Permutations:

//********************************************************************
//
//
//                             Permutations
//
//
//********************************************************************
 
 long Factorial(ushort n)
   {
    long res=1;
    for (int i=2;i<=n;i++)
       res=res*i;
    return(res);
   }
 
 int FindPermutation(ushort elem, int& perm[])
    {
     int res=0;
     int rr;
     int positions[30];
     for (int i=0;i<ArrayRange(positions,0);i++)
        positions[i]=i+1;
     for (int e=1;e<elem;e++)//we don`t look for the last element , there is one seat left
        {
         rr=0;
         for (int i=0;i<elem;i++)
            {
             if (perm[e-1]==i+1)
               {
                res=res+rr;//has already -1, it`s zero-based
                //Print("Found element ",perm[e-1]," to be on perm[",i,"]. Adding ",rr, " to res, equals ",res);
                if (e!=elem-1)
                  res=res*(elem-e);
                //Print("Multiplicating by elem-e (",elem-e,") = ",res);                
                positions[i]=-1;
               }
             else
               { //if the searched element is not on current element,
                 //increment place counter only current position was not previously used;
                if (positions[i]!=-1)
                  rr++;
               }
            }
        }//for (int e=1;i<=elem;i++)
      return(res);  
    }
 
void GeneratePermutation(int number,int elem,int &perm[])
    {    
    string elemallow; 
    int stoppoint,crtpoint;       
    ushort AscX=StringGetCharacter("X",0);
    int i,j,base;
    int nnow;
    elemallow=Replicate("a",20);
    base=2;    
    nnow=number;    
    for (i=elem-2;i>=0;i--)
       {
       perm[i]=Round(MathMod(nnow,base));       
       nnow=MathDiv(nnow,base);
       base=base+1;        
       }
    perm[0]=perm[0]+1;//because first element is from 1 to elem, not 0 to elem-1    
    for (i=0;i<=elem-2;i++)
       {
       if (i==0)
         {
          StringSetCharacter(elemallow,perm[0]-1,AscX);
         }             
       else
          {
          stoppoint=perm[i]+1;          
          crtpoint=0;
          for (j=0;j<elem;j++)
             {             
             if (StringGetCharacter(elemallow,j)==AscX)
                continue;
             else
                {
                crtpoint=crtpoint+1;
                if (crtpoint==stoppoint)
                   {
                   perm[i]=j+1;//write new perm[i]
                   StringSetCharacter(elemallow,j,AscX);
                   }                   
                }
             }//for (j=0;j
          }//else if (i=0)    
        }//for (i=0;i
    for (i=1;i<=elem;i++)
       {
       if (StringGetCharacter(elemallow,i-1)!=AscX)
          {          
          perm[elem-1]=i;          
          break;
          }
       }        
    return;  
    }

This time we got lucky. Just two simple functions to call. Won’t be the case for Combinations.

Combinations

The combinations algorithm is way harder to understand than the permutations algorithm. The most used algorithm is the one that generates combinations in a recursive manner.

Let’s look at a simple combination queue: C(5,3)

1, 2, 3
1, 2, 4
1, 2, 5
1, 3, 4
1, 3, 5
1, 4, 5

First of all, every element has a different freedom degree. For instance, first can’t be beyond n-k+1.
Another thing, is that after you generate the first element, what you have to do then are C(n-1,k-1).
For instance, C(4,2) are:

1, 2
1, 3
1, 4
2, 3
2, 4
3, 4

For each instance of these , C(3,1) are:

1
2
3

Now you can see how it integrates:

When the procedure is called to generate C(5,3) it starts enumerating the first element, from 1 to 3. Thus, it has:

1,(1+x)

Then it calls itself to generate C(4,2) , starts enumerating from 1 to 2.
1,(1+1),(2+x)

Finally calls itself to generate C(3,1) , and completes:
1,2,(2+1) = 1,2,3
1,2,(2+2) = 1,2,4
1,2,(2+3) = 1,2,5

Returns to the previous cycle and continues with C(4,2) with the following element, 2:
1,(1+2),(1+2+x)

Starts calling C(3,1) and completes
1,3,4
1,3,5

Then turns back to continue C(5,3) with the following element, 2.

The algorithm continues until first function call ends the enumeration loop.

The implementation of the algorithm is quite murky. This time we opted for a class. Because, since direct generation is not possible, it’s not feasible to generate all the combinations every time you want a certain combination, in the case when you may want some from a list. So, a class is needed, to provide an inner loop generating the combinations while calling a callback function to pass the user the found combinations, with the option of early terminating.

//********************************************************************
//
//
//                             Combinations
//
//
//********************************************************************
 
 
long CombinationsTotal(ushort n,ushort k)
  {
   return(Factorial(n)/(Factorial(k)*Factorial(n-k)));
  }
 
class CombinationObject
  {   
   protected: 
   long counter;
   ushort n,k;
   bool terminate;   
   ushort current_combination[];
   private:
   ushort last_combination[];
   bool found;
 
   long CombinationsTotal(ushort n,ushort k);
   string CombImage(ushort &ccomb[],ushort n);
   void GenCombinationsRec(ushort depth,ushort nn,ushort kk);
 
   public:
   void SetupCombinationObject(ushort nn,ushort kk); 
   void LoopCombinations();
   virtual void CombinationsCallback();
   CombinationObject() {SetupCombinationObject(6,3);};   
  };
 
void CombinationObject::CombinationsCallback()
  {
   Print(CombImage(current_combination,k));
  }
 
void CombinationObject::SetupCombinationObject(ushort nn,ushort kk)
  {
   n=nn;
   k=kk;
   terminate=false;
   ArrayInitialize(current_combination,0);
   ArrayInitialize(last_combination,0);
   ArrayResize(current_combination,n);
   ArrayResize(last_combination,n);
   counter=0;
  }
 
string CombinationObject::CombImage(ushort &ccomb[],ushort n)
  {
   string res="";
   for (int i=0;i<n;i++)
      res=res+DoubleToString(ccomb[i],0)+" ";
   return(res);
  }
 
void CombinationObject::GenCombinationsRec(ushort depth,ushort nn, ushort kk)
  {
   bool okay,r;
   if (terminate==true)
     return;
 
   if (depth==k)
     return;  
 
   if (kk==0)
     return;
 
   int allows=nn-kk+1; 
   for (ushort i=1;i<=allows;i++)
      {
       if (terminate==true)
          return;
       if (depth==0)
         current_combination[depth]=i;
       else
         current_combination[depth]=current_combination[depth-1]+i;
 
       GenCombinationsRec(depth+1,nn-i,kk-1);
 
       okay=false;//checking for identical combination
       for (int j=1;j<=k;j++)
          {
           if (current_combination[j-1]!=last_combination[j-1])
             okay=true;//okay, new one
          }
       if (okay==false)
         continue;
       else
         { //copying current to last
          for (int j=1;j<=k;j++)
             last_combination[j-1]=current_combination[j-1];
         }
 
       CombinationsCallback();
       if (terminate==true)
         return;      
       counter=counter+1;
 
      }//for (int i=1;i<=allows;i++)
   return;
  }
 
void CombinationObject::LoopCombinations()
  {
   GenCombinationsRec(0,n,k);
   return;
  }

This is a way tougher nut to crack. The interesting function here, the engine itself, is GenCombinationsRec(). The depth parameter tells the function on which level it starts generating the combinations. When depth is depleted, that is, at every end of subsequent call queue, a new combination is generated in current_combination[] and counter is increased. What the user has to do, is to inherit this class in a custom class and override the CombinationsCallback(). Also, the terminate variable is available, if that is set to true, once the CombinationsCallback() ends, all GenCombinationsRec() calls will be early terminated and program will resume execution.

Arrangements

Since arrangements are permutated combinations, there is no point in overcomplicating the issue. The class is almost a carbon copy of the CombinationObject class, with the difference that the recursive method here, GenArrangementsRec(), which is quite like GenCombinationsRec(), generates combinations, and then, when every one of them is ready, permutates it and serves every arrangement (permutated combination) to the ArrangementsCallback(). Simple and convenient, yet results are not in the sorted manner like with Permutations and Combinations:

//********************************************************************
//
//
//                             Arrangements
//
//
//********************************************************************  
 
long ArrangementsTotal(ushort n,ushort k)
  {
   return(Factorial(n)/Factorial(n-k));
  }
 
class ArrangementObject
  {
   protected: 
   long counter;
   ushort n,k;
   ushort current_arrangement[];
   bool terminate;  
   private:
   ushort current_combination[];  
   ushort last_combination[];     
   bool found;
 
   long ArrangementsTotal(ushort n,ushort k);
   string ArrgImage(ushort &ccomb[],ushort n);
   void GenArrangementsRec(ushort depth,ushort nn,ushort kk);
 
 
   public:
   void SetupArrangementObject(ushort nn,ushort kk); 
   void LoopArrangements();
   virtual void ArrangementsCallback();
   ArrangementObject() {SetupArrangementObject(6,3);};   
  };
 
void ArrangementObject::ArrangementsCallback()
  {
   Print(ArrgImage(current_arrangement,k));
  }
 
void ArrangementObject::SetupArrangementObject(ushort nn,ushort kk)
  {
   n=nn;
   k=kk;
   terminate=false;   
   ArrayInitialize(current_arrangement,0);
   ArrayInitialize(current_combination,0);
   ArrayInitialize(last_combination,0);
   ArrayResize(current_combination,n);
   ArrayResize(current_arrangement,n);
   ArrayResize(last_combination,n);   
   counter=0;
  }
 
 
 
string ArrangementObject::ArrgImage(ushort &ccomb[],ushort n)
  {
   string res="";
   for (int i=0;i<n;i++)
      res=res+DoubleToString(ccomb[i],0)+" ";
   return(res);
  }
 
void ArrangementObject::GenArrangementsRec(ushort depth,ushort nn, ushort kk)
  {
   int perm[30];
   bool okay,r;
   if (terminate==true)
     return;
 
   if (depth==k)
     return;  
 
   if (kk==0)
     return;
 
   int allows=nn-kk+1; 
   for (ushort i=1;i<=allows;i++)
      {
       if (terminate==true)
          return;
       if (depth==0)
         current_combination[depth]=i;
       else
         current_combination[depth]=current_combination[depth-1]+i;
 
       GenArrangementsRec(depth+1,nn-i,kk-1);
 
       okay=false;//checking for identical combination
       for (int j=1;j<=k;j++)
          {
           if (current_combination[j-1]!=last_combination[j-1])
             okay=true;//okay, new one
          }
       if (okay==false)
         continue;
       else
         { //copying current to last
          for (int j=1;j<=k;j++)
             last_combination[j-1]=current_combination[j-1];
         }
 
 
       //here combination is final and arrangements are generated       
       for (int iperm=0;iperm<Factorial(k);iperm++)
          {
           GeneratePermutation(iperm,k,perm);           
           for (int jperm=0;jperm<k;jperm++)
              {
               current_arrangement[jperm]=current_combination[perm[jperm]-1];                              
              }                      
           ArrangementsCallback();                        
           if (terminate==true)
             return;      
           counter=counter+1;   
          }   
 
       }//for (int i=1;i<=allows;i++)
   return;                   
  }
 
void ArrangementObject::LoopArrangements()
  {
   GenArrangementsRec(0,n,k);
   return;
  }

And now a sample script (combtest.mq5). The script will display arrangements of n forex pairs taken as k, with a rising k, from 3 to n. Forex pairs are extracted at the beginning, by interogating each tradeable instrument about its calculation mode and retaining only the ones answering SYMBOL_CALC_MODE_FOREX , with the specification that symbol has to have 6 or more letters (two forex pairs and a suffix). Pairs are extracted and added to an array. Then a customized arrangements engine is called to display arrangements:

//+------------------------------------------------------------------+
//|                                                     combtest.mq5 |
//|                                       Copyright Bogdan Caramalac |
//|                                           http://mqlmagazine.com |
//+------------------------------------------------------------------+
#property copyright "Copyright Bogdan Caramalac"
#property link      "http://mqlmagazine.com"
#property version   "1.00"
 
 
#include 
 
int CurrenciesCount;
int PairsCount;
string ForexPairs[100];
 
//our class: we needed it to override ArrangementsCallback;
class LocalArrgClass : public ArrangementObject
  { 
    void ArrangementsCallback()
     {
      string image="";
      image=DoubleToString(counter,0)+" :";
      for (int i=0;i<k;i++)         
         image=image+" "+ForexPairs[current_arrangement[i]-1];
      Print(image);      
     }   
  };
 
//the arrangment object
LocalArrgClass myarrg;
 
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
 
void CheckAndAddPair(string pair)
  {
   bool found;
   if (PairsCount==0)
     {
      ForexPairs[PairsCount]=pair;
      PairsCount++;
     }
   else
     {
      found=false;
      for (int i=0;i<PairsCount;i++)
         {
          if (ForexPairs[i]==pair)
            {
             found=true;
             break;
            }          
         }
      if (found==false)
        {
         ForexPairs[PairsCount]=pair;
         PairsCount++;               
        } 
     } 
   return;
  }
 
void OnStart()
  {  
   string crtsymbol,pair1,pair2;
   CurrenciesCount=0;
   PairsCount=0;
   for (int i=0;i<SymbolsTotal(false);i++)
      {
       crtsymbol=SymbolName(i,false);
       if (SymbolInfoInteger(crtsymbol,SYMBOL_TRADE_CALC_MODE)==SYMBOL_CALC_MODE_FOREX&&StringLen(crtsymbol)>=6)
         {          
          pair1=StringSubstr(crtsymbol,0,3);          
          CheckAndAddPair(pair1);
          pair2=StringSubstr(crtsymbol,3,3);         
          CheckAndAddPair(pair2);
         }
      }//for (int i=0;i
    for (int iarrg=3;iarrg<=PairsCount;iarrg++)
       {
        myarrg.SetupArrangementObject(PairsCount,iarrg);
        myarrg.LoopArrangements();      
       }
  }
//+------------------------------------------------------------------+

Here’s a peek on how it looks (though you have to stop it using terminate – otherwise it will jump some of the log):

You might ask why did we opted for permutations to use int instead of uint or long. The answer is the MathMod() and MathRound() issue. These functions don’t work with integer types. Even if they expect and return an integers as meaning they still work with double as type. So any convertions have to rely on a force cast of a double, which is a signed real type, as int is a signed integer type. Using unsigned integers or even longer long types that span over 8 bytes instead of just 6 like double would have overcomplicated the issue. I think these functions have to be re-engineered by MetaQuotes, to work only with integer types and to abide the received integer types that play the operands, just as div and mod work in Pascal implementations. And as for MathRound() , this must be a real rounding function that returns in the integer type that is to receive the result, or in a generic integer type, if it is used in expressions.

File links:
Combinatorics.mqh
combtest.mq5

A Statistical Method for Self-Tuning Trading Systems

Bogdan Caramalac, MQLmagazine sr.editor — Mon, 24 May 2010 21:28:30 +0000

[Versiunea romaneasca] [MQLmagazine.com in romana] [English edition]

I am starting to write this article in the idea that it might become useful in a not too far time horizon. I have been thinking at this for some time. The autotuning is an idea that is out there for some while, however the extreme complexity of retail trading systems approach made it quite inapplicable. However, HFT trading systems are generally simple, and quite similar rules are applied to hundreds of equities, in a parametrized fashion. Thus, if rules are simple and can be written in “when this, do that , resulting this” instead of maze-like logics of retail systems, then a set of optimal statistics on the entry data (spanning over an array of quotes, orderbook data and so on) is the premise for simple operations like quoting, getting filled, taking profit or stop loss in just a few ticks away. As seen in the previous Progress Apama article, these algorithms are highly unstable, therefore there is a constant run for newer algorithms.

Now you may ask, why would we attempt autotuning of trading systems if the institutionals do not (at least on the HFT level). Thing is, some of their algos are so fast that they might not even have time enough to record the data in a database, so automatic tuning, that requires extra analysis, might be a complete waste of time, with a too small improvement brought to the algos compared to the extra latency that is forming up. What will be coming in these articles has therefore an experimental nature and doesn’t have to be taken as a solution, rather an attempt – I cannot tell for sure if there is any point in attempting to find, with regular statistics, recurring parameters of a stochastic process.

So, let’s presume that an array of parameters (such as individual volatilities, moving averages, executions volume, etc.), is to be fed to a database or to an inner storing array. We have to take into account the fact that a trading signal may exist permanently, at every tick. Supposing the volume to be constant, then same signal can be wired into two actions: either buy or sell. Each signal will therefore have two results, one for buying, one for selling. We could however create separate rules for selling.

For instance, a signal comprised of a volatility and the distance between mid quote and mid quotes moving average comes with: 0.03 stddev & 2 ticks. To twist it a bit, we introduce quoting, and we note 0 for no fill. Wired into a buy with a 3 tick take profit and 2 tick stop loss, at a given time, makes 2 ticks profit, wired into a sell it is not filled. So the buys database will be added a (0.03 ; 2 ; 2) and the sells (0.03 ; 2 ; 0).

And now the things become interesting. At the end, after the data is gathered for a mass of signals that make up the “big number” for the statistics, we have to look thru the database. First, a simple watch will group the database by action and result. In this case, we have six final states:

1. BUY: quote, get filled, take profit (BQFT)
2. BUY: quote, get filled, stop loss (BQFS)
3. BUY: quote, don’t get filled, cancel order (BQC)
4. SELL: quote, get filled, take profit (SQFT)
5. SELL: quote, get filled, stop loss (SQFS)
6. SELL: quote, don’t get filled, cancel order (SQC)

So we have six cases. Analysing BUY with quote, get filled, take profit, is just one of the six states and will comprise a large part of data, the same as the other states. The purpose of the statistical analysis is to have a statistic prepaired per each case. For instance, our case should look, in the end, like:

0.03, 2 : 45% BQFT, 35% BQSL, 20% BQC ; 30% SQFT, 30% SQSL, 40% SQC

After the intial mass of signals along with results is collected, statistics have to be deployed for each combination of the parameters. Having real numbers as parameters, with lots of digits , will lead to the fact that you may have actually a number of cases equal to the number of signals that were the input, rendering statistics impossible. So, there has to be applied some data aggregation. For instance, for each parameter set, can be calculated the quartiles then the number of bins with the Freedman-Diaconis formula (this is an example, other methods can be applied). This will yield up a different number of bins for each of the parameters. Then a new array will be created, where each case will have a sticker denoting its number of bins. For instance the 0.03 stddev is labeled as “Bin 1″ on the stddev scale and the 2 tick distance is labeled as “Bin 2″ on the tick distance scale. The total number of cases may be smaller or equal than the product of bins count per each criteria. If you have 3 criterias and 10 bins each, that may be even 1000 number of cases. More likely, a core number of cases will have a lot of situations recorded, while others will barely have one situation recorded. However, the more platykurtic the repartition of situations per cases, the more enforced the need for a “bigger number” of situations. After applying the statistics, our database statistics will look like:

Bin 1, Bin 2 : 45% BQFT, 35% BQSL, 20% BQC ; 30% SQFT, 30% SQSL, 40% SQC

What is important is that relevant statistics is to be extracted. If this is the current case, what would be the trading decision? Go for a buy with a chance of take profit in 45% of the situations this case appears? Only that it has to stand to 35% stop losses… Because on the sell side it’s even worse, with stop losses situations roughly equal to take profit ones. This case has to be decided as a buy case, although with reserves. Unless the profits are relevant enough, there is no decision to be taken. Nevertheless, it is mandatory for the algo to have a decision module that picks only relevant cases for trading. But how fast can the machine be, because after the initial data is gathered, each new signal, (that can be even a new tick, in a HFT-like algo), will trigger a massive recalculation of bin sizes on the entire database, to update the statistics, and at the same time to allow the delivering and execution of the trading decision in real time?

Remember my article about the CEP engine. The CEP is visioned there as a recognition – trigger module, but what if the CEP is first used as a recording device ? What if the CEP is wired to the statistics engine? How much latency would this add to the calculus? Can we still talk about real time?

Progress Apama : How Deep Does HFT’s Rabbit Hole Go ?

Bogdan Caramalac, MQLmagazine sr.editor — Wed, 31 Mar 2010 19:34:46 +0000

[Versiunea romaneasca] [MQLmagazine.com in romana] [English edition]

This month, while waiting for the new Strategy Tester, I felt myself pretty void of ideas. I realised that the content that we produce is effectively drying out. The backtest is the creed of the trader, what gives him confidence in the trading strategies that he attempts.

By the mid of march I stumble upon some Progress Apama recorded webinars entitled “Build Quickly, Run Fast” and “From Concept to Profit in No Time Flat”. Sure, our study objective is not the Progress Apama platform. That is an institutional, high frequency platform, that we’ll never put our hands on, however it’s interesting to see more of their approach on HFT, as well as some of their concepts in both building and running. Not even big market players use it, but a very tiny elite of them, la crème de la crème, since they have about 120 clients globally.

Problem, is HFT is becoming mainstream in institutional trading. And when the institutionals are doing something, better open the ears wider, cause they’re always getting all the money and they’re probably right!According to a Tabb Group research, HFT firms in US are just 2% of 20K trading firms, and make up 73% of the total equity volume traded. It’s looking like the entire equity market is under a “hostile takeover” from HFT, and the process will not stop. More and more trading firms are getting new trading technologies, or, if not possible, at least new HFT-like algos to run on their own platforms, so this trend will not stop. It is important that the retail environment remains aware of what is out there, so it would not have the market perception completely distorted by paradigms too old to fit the current developments.

It seems that , from the beginning of markets, traders always struggled for latency, to be the first in getting a certain trade. The first example that comes in my mind is Nathan Rothschild, that found out the first that Napoleon was defeated at Waterloo and lead the entire market into believing that Napoleon was victorious to crash it enough for him to buy it a pound for a penny. Another example more close to our days is the character from “Reminiscences of a stock operator” who loses his grip on the market and cuts his latency by using the newly technology of the time, the telex. With every major innovation in telecom and computing, a new layer was added, and what it was in years before the known level II markets, where human traders were manually placing and executing trades, is now the realm of high frequency trading machines. At first, it was all about seconds – as it is the case for fellow MetaTrader folks – then they started going deeper and deeper to milliseconds, microseconds, and even nanoseconds, that they think about right now. This happened because HFT technologies adapted, and modified market behaviours on every time layer : when milliseconds started to become available, the algos running on slower stations to become faulty, now, when it is microseconds time, the current millisecond-based stations are in the process of upgrading, and tomorrow, nanosecond-based stations will replace the ones working on millisecond level.

Let me be sarcastic here a bit: a microsecond is 10^-6 of a second, and a nanosecond is 10^-9. There is a very long technology road towards the Planck time, which is supposed to be the minimum amount of time: 5.4 × 10^-44 s. At the current rate, we should expect latency being cut by 2 powers of 10 by year, unless something really happens to change the curvature of latency cut.

What happens to algorithms due to improvements in execution and expansion of HFT desks? As Louis Lovas, Chief Architect for Progress Apama, explains “it becomes either obsolete , reverse engineered, or the spreads become so tight that is no longer profitable”

Dan Hubscher, Principal Product Marketing Manager – Capital Markets for Progress Apama citing an AITE Group research, says that “even in a relatively stable environment, the lifespan of an algorithm can be as short as three months”. This was the reason for developing the Progress Apama Studio, a rapid strategy development environment. Because if the lifetime of an algo is limited, it’s logical that if you cut it’s development time you get extra profits by running that system for a longer time, with the extra time gained when others are struggling to replicate it on other platforms with worser development environments.

The following is an example from their demo. The strategy is a Statistical Arbitrage between two stocks. A mean is calculated , along with Bollinger Bands at 1.8 standard deviations. The following screen depicts the strategy running in one selected instance (Bank of America vs Citigroup) , and the Scenario Instances window below shows what is the algo doing on each instance (first line is Caterpillar vs Chevron, second is Bank of America vs Citigroup). It is a predefined strategy, it comes as a template with the station, there is nothing secretive and important about it.

A few notes about the strategy.
1. As you can see, only one spread was set up to be displayed – and you’ll see below, strangely, they picked the spread between mid prices. Normally, if the strategy would be based on market executions, you have at least two spreads to monitor, and the exit criteria would use the spread which is opposite to the one taken ; if the strategy would be based on quoting, there are four spreads to monitor.
2. The quantities are predetermined, as parameters, first is 10, second is 20. The algo doesn’t adjust them by hedge ratios.
3. I can see no tick count or time span of ticks used for average and Bollinger Band calculus.
4. The max quantity fields are used for position control. Yes, that hated position that MT5 has too now. If you look at Scenario Instances, in the Current Position fields, you can see that they are not 10 and 20, it’s not having only these quantities at a time, rather positions are bulked up with new and new orders of 10 and 20, until closed or touch the max quantity parameter. Such order bulking may appear if one of the Bollinger Bars is crossed repeatedly without the touch of the other bar.
5. The order timeout seems to have the role of a position decay, positions being flattened after 20 sec. It can’t be an execution timeout because a second is a century now in HFT, and waiting 20 sec for an order confirmation doesn’t sound to me HFT at all. Or was I wrong about HFT ? See the article about MetaTrader , Tick Data & Charting : Why Not a Level II MetaTrader ? !
6. Note the time scale of the chart. There are 12 seconds between time labels, and there seem to be from 3 to 7 executions per interval. Know any EA trading like that?

What is interesting in Progress Apama is how strategies are built. Strategies can be built even without coding , but unlike environments in other fields, that attempt building without coding, Progress Apama comes with a complex, yet simple to use concept. As Dan Hubscher says, “according to AITE Group, the amount of time it takes to develop a new algorithm, from data acquisition and analysis, to strategy deployment, and thru to production is typically 10 to 28 weeks”. So it was quite natural to come with rapid application environment, because this is the problem that hits other direct competitors on the institutional platform market, and put Progress Apama into a more favorable position. In their development system, all variables that have to be either directly watched, or calculated, originate from smart blocks, in a block diagram. Each block takes data , either from the market or from another block, and outputs it as an input for another block, generating at the same time internal variables (which you can see in the green panel on the right).

Then the strategy is designed on a workflow basis, kinda similar to the diagrams used when learning algorithmics. Each block contains conditions and commands written in plain english, all conditions , commands and variables being selectable from a menu appearing upon right clicking.

As you can see, it has a pretty fancy development environment, that has to answer quick to the ever expanding ideas and complexity degree of trading algorithms, but the real power of the station unleashes when running the strategies. This is what makes it a top notch platform.

The Statistical Arbitrage example is quite simple, as it follows the spread as its driver. But real HFT strategies are more complicated, each strategy may be following asynchronous events that happen on multiple assets:

When did we ever have such an approach ? When did it matter the time restraint in which the signals were supposed to come? Oh, I forgot, we didn’t even have the backtester…
An interesting thing to note about their view, is the WHEN approach. Repeated times, Dan Hubscher and Louis Lovas explain that WHEN is not the same as IF. No comment.

To make this kind of conditions usable for the strategy designer, Progress Apama comes with a smart thing called Complex Event Processing:

Dan Hubscher defines Complex Event Processing as “the ability to relate two or more events, even if these events occur in a span of time, and then determining meaning from that relationship immediately, such as detecting a trading signal, or an alpha-generating profit opportunity”.

But we don’t even have OnNews(), ain’t that right?…

It’s clear, that “immediately” word there means that you must have the lowest possible latency. I believe that the CEP can be translated into MetaTrader, by starting from OnChart() and OnTrade() implementations, like the ones from my past articles, and transforming these methods into Complex Event Processors, that translate tick and trade events into intelligible events, such as “MSFT above 10-second Bollinger Band”, followed by a time stamp, and put into a queue. The queue might be then taken over by another procedure, that applies the time constraints, generates signals, and passes them to decision modules. Check our article Anatomy of a simple CEP engine.

However, the thing that makes Progress Apama the million buck machine is none of these things, because these are simple tools, that can be used by retail stations too, if their developers would include such facilities. Sure, the rapid development tools matter only when compared with competitors. However, it is the ability to run these strategies in hundredths of instancies, taking advantage of multicore infrastructures, without losing the latency advantage. And this is done by Event Correlator(s).

Progress Apama has also a programming feature. They come with the proprietary Monitor Script language, a Java-like language, although simpler, side by side with a full Java implementation:

As I said above, I include below parts of the Statistical Arbitrage code , the ones that explain the core of the strategy, found in the webinar:

This is the beginTrading() action, that quite shows the candies of MonitorScript. As you can see, the route in this case, are getting the data from the market. Bust most important here, is the on statement. This statement defines listeners, having a syntax like:

1	on [all] <event expression> <action part>

The first on that you see checks for the destruction of the instance by the user (in a similar manner to WM_CLOSE being sent to WindowProc() ), then it removes the variables and closes the instance. The secondary on is more interesting. Here, the second and the third one are written in a quite redundant manner, because the first condition which is an and is replicated inside the code, with an or. As Louis Lovas notes in the comment , “Make sure we have at least one tick from each before moving on” it might appear that the Depth() event is triggered by a new tick, but it actually seems that the Depth() event is triggered when the structure of the orderbook changes, meaningly when at least one volume changes on either the Bid or the Ask side, not only when a tick comes (change in core Bid/Ask). Depth() seems to trigger similarly to the OnBookEvent() that was not yet completed by MetaQuotes, which I explained in other articles that should trigger way more frequent than OnTick(). Leaving this comparison aside and continuing with the code, see the second condition, which is written by Louis just to show how flexible is the syntax, and then, what it triggers, the processNextSpread() action.
An interesting thing is to be remarked about listeners. By looking at the listeners we see a very advanced syntax, way beyond the rigidity of the event-driven system that MetaQuotes is struggling to produce, and which exists in other stations. First, the listeners apply to every kind of event that is linked to subscribed data. Beyond this, the event description appears in the listener condition, and it can be anything. Even if you would like to get a tick on another instrument in MT5 you need to run a separate EA (daca nu cumva va deveni posibil prin OnBookEvent() ) because OnTick() answers only to current instrument. At the same time, each listener is event condition – action implementation, deployable quite everywhere inside the code, giving a high degree of freedom to the programmer, whereas in the event driven model, the treating for every event type is confined within a single code body, where it analyses all the possible events of that type, which, if translated in MonitorScript, would mean that all listeners working with a certain event type would have to reside in a single code body.

As you can see, the spread is really calculated with the mid prices. What’s more important, is that the method incapsulates all the strategy itself, and you can see how it sends orders, without checking previous orders, thus demonstrating once again the positional system.

Seems the entire SubmitOrder() action has been written with the purpose of updating strategy variables. Probably the entire code could have rested inside processNextSpread(). Strangely, the position variables are calculated , by adding or subtracting volume, instead of simply using a function to retrieve the position per each instrument, in the parts of the code where this is necessary. What is not clear in the code is where the maximum position constraints apply, as well as the order timeout.

What is essential about Progress Apama is that it can run millions rules concurrently and make use of multiple cores environments.

Here you can see a strategy with simulated data running in 500 instances on a quad core machine:

So , if Louis Lovas says that it works on a quad core machine, and we don’t have any reason not to believe it, means that Irene Aldridge was right, and gamers ever increasing requirements pushed the hardware to the point that machines able to run HFT are in every home ; but , which broker will give you low latency for your poor retail account and where will you get from a platform that mustn’t waste processor resources?

Of course, we will never be able to do that. MetaTrader folks don’t even have the cash to meet the margin requirements of running portfolios that rebuild indexes with required weights, so low latency trading is out of discussion. MetaTrader will never ever have low latency. However, some high frequency trading elements, such as algos and complex event processing for low scaled strategies, that span on a very limited number of assets, can be ported there. The purpose of this article was to introduce readers in the HFT world and its specific issues.

Tick Data & Charting : Why Not a Level II MetaTrader ?

Bogdan Caramalac, MQLmagazine sr.editor — Wed, 31 Mar 2010 19:34:35 +0000

[Versiunea romaneasca] [MQLmagazine.com in romana] [English edition]

You should view this article as a continuation of the one about Progress Apama, because it brings institutional issues to MetaTrader5. If you look at that Statistical Arbitrage strategy, you’ll see it doesn’t have but an execution at 1-2 seconds at most. So, that doesn’t seem HFT at all. But becomes HFT in the context of running this in 500 instances – because it will have hundreds of executions per second. However, if you keep up to 10-20 instances, is something that MetaTrader5 can handle.

That strategy inspired me to think about an indicator that draws tick charts. Sure, you can view a tick chart, but in the Market Watch, by clicking on “Tick Chart” near “Symbols”. But what if you want to see more than one at a time? So I wrote an indicator to do it. However I did it quite poor, and Rosh (Rashid Umarov) came to the rescue. This is the indicator body, modified after inserting levels:

//+------------------------------------------------------------------+
//|                                                        Ticks.mq5 |
//|                        Copyright 2010, MetaQuotes Software Corp. |
//|                                              http://www.mql5.com |
//+------------------------------------------------------------------+
#property copyright "2010, MetaQuotes Software Corp."
#property link      "http://www.mql5.com"
#property version   "1.00"
#property indicator_separate_window
#property indicator_buffers 2
#property indicator_plots   2
//--- plot Bid
#property indicator_label1  "Bid"
#property indicator_type1   DRAW_LINE
#property indicator_color1  Blue
#property indicator_style1  STYLE_SOLID
#property indicator_width1  1
//--- plot Ask
#property indicator_label2  "Ask"
#property indicator_type2   DRAW_LINE
#property indicator_color2  Red
#property indicator_style2  STYLE_SOLID
#property indicator_width2  1
#property indicator_level0  0.00
#property indicator_level1  0.00
#property indicator_level2  0.00
#property indicator_level3  0.00
#property indicator_level4  0.00
#property indicator_level5  0.00
#property indicator_level6  0.00
#property indicator_level7  0.00
#property indicator_level8  0.00
#property indicator_level9  0.00
#property indicator_level10  0.00
#property indicator_level11  0.00
#property indicator_level12  0.00
#property indicator_level13  0.00
#property indicator_level14  0.00
#property indicator_level15  0.00
//--- input parameters
input int      number_of_ticks=1000;
input int      points_indent=10;
input int      LevelsCount=7;
//--- indicator buffers
double         BidBuffer[];
double         AskBuffer[];
//+------------------------------------------------------------------+
//| Custom indicator initialization function                         |
//+------------------------------------------------------------------+
int OnInit()
  {
//--- indicator buffers mapping
   SetIndexBuffer(0,BidBuffer,INDICATOR_DATA);
   SetIndexBuffer(1,AskBuffer,INDICATOR_DATA);
   PlotIndexSetDouble(0,PLOT_EMPTY_VALUE,0);
   PlotIndexSetDouble(1,PLOT_EMPTY_VALUE,0);
   IndicatorSetString(INDICATOR_SHORTNAME,Symbol()+" Tick Chart");
//---
   return(0);
  }
//+------------------------------------------------------------------+
//| Custom indicator iteration function                              |
//+------------------------------------------------------------------+
int OnCalculate(const int rates_total,
                const int prev_calculated,
                const datetime &time[],
                const double &open[],
                const double &high[],
                const double &low[],
                const double &close[],
                const long &tick_volume[],
                const long &volume[],
                const int &spread[])
  {
   static int ticks=0;
//---
   if(ticks==0)
     {
      ArrayInitialize(AskBuffer,0);
      ArrayInitialize(BidBuffer,0);
     }
   setMaxMinPrice(ticks,points_indent);
//--- CopyRates
   MqlTick last_tick;
   if(SymbolInfoTick(Symbol(),last_tick))
     {
      BidBuffer[ticks]=last_tick.bid;
      AskBuffer[ticks]=last_tick.ask;
      int shift=rates_total-1-ticks;
      ticks++;
      BidBuffer[rates_total-1]=last_tick.bid;
      AskBuffer[rates_total-1]=last_tick.ask;
      PlotIndexSetInteger(0,PLOT_SHIFT,shift);
      PlotIndexSetInteger(1,PLOT_SHIFT,shift);
      Comment("Bid =",last_tick.bid,"   Ask =",last_tick.ask);
     }
 
//--- return value of prev_calculated for next call
   return(rates_total);
  }
//+------------------------------------------------------------------+
//| set Maximum and Minimum for an indicator window based on last values
//+------------------------------------------------------------------+
void setMaxMinPrice(int last_values,int indent)
  {
   int dgs;
   double dist;
   int visiblebars=(int)ChartGetInteger(0,CHART_VISIBLE_BARS);
   int depth=MathMin(last_values,visiblebars);
   int startindex=last_values-depth;
   if (startindex<0) startindex=0;
   int max_index=ArrayMaximum(AskBuffer,startindex,depth);
   max_index=max_index>=0?max_index:0;
   int min_index=ArrayMinimum(BidBuffer,startindex,depth);
   min_index=min_index>=0?min_index:0;
   double MaxPrice=AskBuffer[max_index]+indent*_Point;
   double MinPrice=BidBuffer[min_index]-indent*_Point;
   IndicatorSetDouble(INDICATOR_MAXIMUM,MaxPrice);
   IndicatorSetDouble(INDICATOR_MINIMUM,MinPrice);
   dgs=SymbolInfoInteger(Symbol(),SYMBOL_DIGITS);
   dist=NormalizeDouble( (MaxPrice-MinPrice)/(LevelsCount+1), dgs);
   for (int i=0;i<LevelsCount;i++)
      {
       IndicatorSetDouble(INDICATOR_LEVELVALUE,i,MinPrice+dist*(i+1));
      } 
  }
//+------------------------------------------------------------------+

As a tick indicator, it looks okay. But what if you want to put something else on it, like Moving Averages, Bollinger Bands, or even monitoring spreads, like in the Progress Apama strategy window? That means you have to write over and over again, newer and newer indicators, just to show you what you need. Sure, tick calculations are easy from inside an EA – they will happen mostly in the OnTick() event – but you need indicators to help you visualize the strategy, even if their practical application will be most of the time unnecesary, due to separate calculus in the EA. But keep in mind that there is no tick history. Such indicators draw from scratch. So if you want to see something longer, at least now, when the Strategy Tester is not available to run the indicator on, you have to give it time to display, it’s not instant as for other indicators.

So, if tick data would be available, meaning PERIOD_TICK also, then some good things for us might appear, as:
a. No need to write special indicators for tick level.
b. Tick level charts appear rightaway and span for entire history.
c. Tick history will make the backtester more accurate, because the backtester doesn’t have to invent ticks to cover such ares.
d. (drawback): Huge hard drive space occupied by history, larger memory requirements.

What does it mean for MetaQuotes:
a. Change in history and data aggregation, starting from tick to make the rest, not from M1, keeping M1 as a data source for old history, where tick data was not available;
b. Light modification to data access functions (fields such as openbid, openask, closebid, closeask in MqlRates etc).
c. Light modifications to the backtester;
d. MetaTrader branded as a near institutional platform.

Why should we want this? Well, why the big boys don’t use bar charts? After all, there are no surprises on tick charts – and we will investigate this some other time – so you won’t fear that a spider leg bar will touch your stop loss in a middle of good looking trend.

Thing is, if you have Level II available, and if you automate it, you are very close to the HFT realm. In fact, what is being blamed today is not the entire algorithmic trading – because they never think about users of MetaTrader, E-Signal or even NeoTicker – but the institutional , unfair (to be read “colocated”) trading. At the surface, the discussion is about computerized vs manual trading. But these , both sides of the conflict, act on the Level II, with the possibility to quote and be filled (i.e. buy at bid, sell at ask), and pay tiny fees for these trades. Thing is, if we’d benefit Level II plus MetaTrader automation, with lower brokerage fees, some forms of HFT would become applicable. This is a thing that probably remained hidden, the fact that low latency trading seems to be not the same thing as high frequency trading, even if the terms are used interchangeably. I believe that algorithmic Level II trading can be designed to be HFT even if it doesn’t benefit from low latency execution. Problem is, traders were indoctrinated so much with Level I, that is very hard to make Level II trading accepted. Because Level I trading has a “buy or sell now, and see later if you were right” mentality, whether Level II has a “see this, quote it, if get filled, close later” mentality. The same “later” has different meanings, Level I “later” means from minutes to up, usually hours, whether Level II later means “seconds” (for normal traders) , or “milliseconds” for hedge fund systems. Trailing stop is tighter on Level II, trails pip by pip. And all is meant to make real time money. Not in a few hours, and not tomorrow or not next week. But rather, see equity going up a few times per minute. This gives a way higher degree of confort, way higher than in Level I, where you can watch losses accumulating creepy for hours or days. What is most often forgot in trading, is that trading strategies are not for robots that can wait an ethernity to improve their trading styles. Life happens in real time. We are not immortal and beyond it, youth is damn short. This is one of the drivers for cutting down the latency. On one hand, it is the need to be filled before the competition, on other hand, more and more profits per a lesser and lesser time unit means fatter salaries for management… While this, even the most rational and successful traders that on Level I cannot exceed some limits of the yield. Essentially, the Level II automated systems have this job, to compress the time, and get similar yields as the ones from Level I, but in a hundredth or lesser time. But these yields are relative and small, when looked upon from inside the hedge fund industry. Because, the system’s results are relative to the average liquidity used. That is, a lot of liquidity will remain unused, but has to return yield. And while diversification is way to use more liquidity, the time span is the other side of the coin, because one trade a second is a thing, and a hundred trades in the same second, spanning lots of microseconds each, is another thing. On another hand, HFT platforms cost shameful amounts of money, and smaller institutions do not afford them, because a lot of the yields from the market go on platform costs and PhD salaries. Now you understand why a Level II MetaTrader, even if not low latency, is important? As Irene Aldridge admits too, for the buy side, there aren’t many companies yet selling products. In fact, if we exclude Strategy Runner, X-Trader and Ninja Trader, I can’t think at others. X-Trader has the automation pretty institutional in terms of costs, and Strategy Runner is so cryptic that you have to pay lots of cash for development. None of these platforms integrate options, however, and making a market in options, getting filled there, is a great way to make economies, as option spreads can be really high.

After all, if look on the MQL5 forum, you can see that MetaQuotes at least seems to commit to make MetaTrader 5 the best trading terminal in the world. If MetaTrader5 becomes Level II ready (remember the old motherboards with the Prescott ready sticker ?) and by generating an increased awareness about the importance of tick level action in the retail trading community, it might determine changes on brokerage levels.
As brokers will give more access to tight spread ECNs, the spreads will go down, and actually the spread based model might become even extinct, replaced by a commission model – and I mean small commissions, small enough for the trader to feel that he trades in an ECN not just a retail feed that brands commission instead of spread. This way a lot of strategies from the institutional levels would become portable to MetaTrader and retail world – which will probably become “institutional with big latency”, or, more important for retail traders, FREE, easy-to-deploy HFT solution that is not low-latency based. The problem is, that commissions are driven by volume. The larger the volume, the smaller the commissions. However, at least forex traders are heavily leveraged. (Yes, the americans will soon not be able to do this anymore). Being highly leveraged, small losses are still hitting the equity hard. How much could you support commission losses until volume becomes high enough to get rebates and jump to profit? Let’s not forget, the global economy is in recession, and brokers, as any regular business in this word, live off sales… Which makes them more open for innovation, offering from low to tiny commissions…

On another hand, how could MT be used as a Level II platform the way it is? Sure, the OrderSend() can be used in a Level II manner, but without tick data, and with a backtester that just models ticks, instead of using ticks for real? I guess not. Tick data is a must if the station would be rebranded to a Level II station.
However, there are things to do until there, as MetaQuotes still has older homeworks to complete : the Strategy Tester, a better OnTrade() syntax, the Depth Of Market view and of course the OnBookEvent(). When all of these will be completed, MT5 will be so far that you’ll forget about the old MT4 candies, I’m sure about that…

But if the retail community won’t ask for more bang for the buck, nothing will be achieved, cause automated Level II is something they are not likely to slip for free.

Anatomy of a simple CEP engine

Bogdan Caramalac, MQLmagazine sr.editor — Wed, 31 Mar 2010 19:34:10 +0000

[Versiunea romaneasca] [MQLmagazine.com in romana] [English edition]

If there wouldn’t be that CEP abbreviation there you’d probably ask how that something inherently complex could become simple. Thing is, everything starts by being simple at first then evolving to something more complicated.

Even if you might have read my Progress Apama article, I remind you here the definition given by Dan Hubscher to CEP : “the ability to relate two or more events, even if these events occur in a span of time, and then determining meaning from that relationship immediately, such as detecting a trading signal, or an alpha-generating profit opportunity”.

So, given that events have to be related, within a time constraint, there must be an event queue, which stores individual events , as they are being received. Sure, storing is not made ad infinitum, in an ever increasing queue ; the queue size has to be dynamic, but it’s length in time should not exceed a predetermined amount of time, say 5 minutes. This is why a CEP engine, that is mainly targetted for HFT and Level II trading, doesn’t have to mixed with Level I trading signals, because these span on a way higher amount of time, lots of minutes to hours or even more, making the event stack not only to grow to an unmanageable size, but also to slow down the engine speed.

The covered amount of time is the first limitation of the engine.

Second, is the link between simple events. Simple events are not fully independent, as the Progress Apama webinar slide might show. For instance, you are considering a cross of higher Bollinger Band as an event, but also the cross of the lower Bollinger Band. These events are completely opposite. Should both happen, at the time the second happens, the first one must elliminated from the queue, as you see in the scheme below that event 4 is deleted. There are two paths that may be chosen, either opposite events are removed from the event queue after some time – which is diffcult to determine, or before final validation of a complex event there are searched opposite simple events coming after the ones that make up the complex event.

Third, avoiding event redundancy. Bad implementations can cause redundancy, meaningly same event to be reported over and over again. For instance, “MSFT is below 5-minute moving average”. A parenthesis here – I don’t mean the 5 minute chart, rather the moving average of all ticks occurred in last 5 minutes, that’s why I said MetaTrader should have tick data and tick period enabled.When it happens, it is an event. But the next second, at the next tick, it will be reported again, and the engine has to reject it, because it already is in the queue, as there is the case with the orange event from the scheme, which doesn’t even gets a number, because it is rejected from the beginning. Such situations should be avoided from start – the engine mustn’t lose time for redundancy check, otherwise the engine should be able to filter for these events , but it’s better that the event stream is cleaned before.

This is the schematics of the engine. The tick events, or the more complex calculus events originate from an OnChart() event that is triggered by more Tick Event EAs, as they are described in the article Tick events in MQL5: complicated for now, but functional .

The first column is event number in the queue, from the most recent to the oldest. Each simple event, that is linked to the data, has a number allocated to it. The same event, happening on a different instrument, has to be given a different number, so same event sequence making up a complex event will be made up of same simple events but with different identfiers. The engine must know which events are completely opposite, for instance 1 and 3, also which event is allowed to appear multiple times in the queue, because some may be allowed (for instance execution events (the ones beginning with E in the scheme)). All other events will be forbidden from appearing multiple times – but however this should not be done by the engine per se, the one that moves the event queue, rather these must be forbidden from the pre-processing phase.

At the same time, the CEP engine must recognize each complex event knowing all simple event arrangements.
For instance, if a complex event for Microsoft (MSFT) is made of “10 followed by (11 or (12 and 13))” within 3 minutes, then the binder will look for, within 3 minutes: 10, 11 ; 10, 12, 13 ; 10, 13, 12 meaning for all alternate paths. If the same event would mapped for SP500 (^GSPC), given that its sub events will have different numbers, the condition might look like “5 followed by (15 or (16 and 17))”, having the arrangements: 5, 15 ; 5, 16, 17 ; 5, 17, 16. Such arrangements can be given directly to the engine, written directly inside, or generated inside with an algorithm, transforming the binary expression, with the notable difference that “FOLLOWED BY” is a non commutative AND. Because the rules are defined as series of events appearing in a certain order, the negation can’t exist as in a binary expression. It can be implemented only a a interdiction for some events to appear over the entire time constraint or within some sequences, as simple forbidden entries between required events.

You can see that I included in the scheme two bind columns. That is , one simple event can be binded to more than one complex event, because otherwise some rules would never trigger. Complex events would have a separate ID set, that is not to intersect with the simple events ID sets. The engine core will send the complex events to be processed by user in the EventsCallback(), but at the same time, the user could pipeline these events to another CEP engine.

The matching algorithm begins with the construction of the simple events matrix per each complex event. The simple event matrix will contain on each row an arrangement. Because checking is done backward, from the later events to older events, arrangements have to be written in the matrix in a reversed manner. Each complex event will need passing thru the events queue only for as long as the time constraint spans, in a vertical manner : first pass will check events on the first column, and simple events found this way will be pre-binded ; second pass will check the second column only for arrangements that had a pre-bind on the first. When no new pre-binds take place, matching is aborted for that specific complex event. However, to minimize the time needed, each pass will regard all events, before jumping to the next pass. When searches are cancelled for all complex events, algorithm ends.
When an arrangement has all simple-events pre-binded, it’s time to check for negations. First, there will be an entire period check for global negations – events forbidden during entire complex event, then for events forbidden between pre-binded events. When all negations are checked,a final check has to be done to ensure they still abide the time constraint, and only in this case, simple events are binded to the complex event, provided that in the bind array of each simple event this bind is not previously done – to avoid re-binding to same complex event. The resulting complex event is then put in a bind queue and passed to EventsCallback(). Sending events to EventsCallback() must take place after the algo is done, otherwise there might appear recursive engine calls that will not end their execution.For the simple events that are being binded, the current bind array position will be filled with the complex event, the position is then incremented by one.

Here is an example of different complex events in different stages of pre-binding:

For the MSFT event, the engine finds event 12 on the first pass, and pre-binds it on the third arrangement. On the second pass, it finds event 13, and pre-binds it to the second arrangement. It finds 12 too, but it can’t pre-bind it to the second arrangement because it occurred before 13. If on the next pass finds the older 10 event, third arrangement is completed, being no negations there, and if time still abides constraint, provided that complex event 1 was not binded yet, then simple events on the third arrangement are binded to complex event 1, bind counter incremented, event placed in the bind queue. For the ^GSPC event (complex event 2), the engine has completed already the first arrangement. Event 5 does not appear checked for second and third arrangement because events on the previous columns are not checked.

As for the execution events, these are sent by the EnumCallbacks() event of the DealHandler class from the Distinguishing quasi simultaneous fills – class to report last deals article. Simple execution events can also be mapped in more complex execution events because deals have magic numbers and comments, and if you buy two times in row same 1000 stocks, you will know which buy was part of which complex execution event. This is why such events could be treated either seperately or in combination with market events.

We will come with a coding example in a future article.

DLL Hell, MQL5 edition : UNICODE vs ANSI

Bogdan Caramalac, MQLmagazine sr.editor — Wed, 31 Mar 2010 19:33:41 +0000

[Versiunea romaneasca] [MQLmagazine.com in romana] [English edition]

Many many years ago, when we were kids, in the beginning years of the crazy 90s, two languages were in battle in developer world. Pascal, with a down-to-earth, easy to understand syntax, well suited to a high level language, and C++, with a more cryptic, but faster to use syntax, well suited to its medium level. C++ won the battle, and everything that was done in Windows became compiled in C++ and beared its marks : null-terminated strings and what was known at that time as standard calling convention.

The null-terminated strings were normal strings, known as ANSI strings, but at that time there was no UNICODE yet. Every character was a single byte and the strings had a dynamic lenght, as they were supposed to end with a null (a zero byte). Thus applications were receiving a pointer to indicate were to read these strings from, and knew where the strings end, by looking for a zero byte. As for the standard calling convention, on procedure call, C++ compiler was pushing parameters on the stack starting from the last and finishing with the first.

Null-terminated string (ANSI)

|---------------|
|c1|c2|....| 0  |
|b1|b2|....|bn+1|
|---------------|

Pascal was the absolute reverse of C++ in all these matters. Strings were ANSI too, one byte per each character, but strings had a fixed length of 255 bytes, or compiler defined. They had an extra byte in the front, specifying the logical length of the string (how many bytes were actually used). As for the calling convention, this was perfectly reversed, as in the pascal calling convention parameters were pushed on the stack from the first to the last.

Standard Pascal string (ANSI)

|------------------|
|ln|c1|c2|....|c255|
|b1|b2|b3|....|b256|
|------------------|

This is why Pascal strings could have been sent entirely to functions, without the need to send by reference, which is the unique mode that string sending is possible in C++.

As C++ won the battle, Pascal compiler had to adapt, and calling convention was an easy task. As for the strings, the problem became complicated, as developers had to struggle with PCHAR, a name given to fixed arrays of one byte per element, which were supposed to hold C++ null-terminated strings and were sent by reference.

As these were not enough for developers, the UNICODE standard came in.

UNICODE is a complicated standard, and I don’t know it entirely. The difference from the ANSI is that UNICODE characters are wider, generally they span on two bytes each, but there are also 4-byte character coded strings. In the beginning, UNICODE strings seemed to be something new and awckward. Thus, they were called wide strings. Windows API Functions working with wide strings were had a name terminated with W ; pointers to null-terminated ANSI strings were char* , thus pointers to null-terminated UNICODE had to be called wchar_t* .

Null-terminated strings (UNICODE)

|------------------------------------------|
|  c1 |  c2 |....|      cn   |      0      |
|b1|b2|b3|b4|....|b 2n-1|b 2n|b 2n+1|b 2n+2|
|------------------------------------------|

MQL5, as most of the programming environments nowadays, is UNICODE. Even simple strings that you use regularly are still UNICODE. They have an ANSI look, but internal representation is UNICODE. This is because ANSI can be packed in UNICODE, filling unneed bytes with 0.

ANSI packaged in UNICODE (MQL5 normal strings)

|------------------------------------------|
|  c1 |  c2 |....|      cn   |      0      |
|b1|0 |b3|0 |....|b 2n-1| 0  |b 2n+1|  0   |
|b1|b2|b3|b4|....|b 2n-1|b 2n|b 2n+1|b 2n+2|
|------------------------------------------|

So, in a UNICODE-packed ANSI, every even byte is 0.
But what if you have an older C++ DLL, who uses null-terminated ANSI strings ?
That means it expects and returns null-terminated ANSI strings.

So, if you are to send an “ABC” string to such a DLL, it have mapped in bytes: 65, 0, 66, 0, 67, 0.
The DLL will see the first 0 as the null terminating the string and will understand only “A” from the entire string.

If you are to receive an “ABC” from this type of DLL, you would receive in bytes: 65, 66, 67, 0.
The UNICODE MQL5 will understand first character as 65 and 66 (making something chinese-like), and the second character as 67 and 0, mapping to “C”. Then it will continue reading, if there is no access violation, until it finds 0 and 0, making up for the null, resulting in a complete jabber. The access violation might be avoided because MT5 might allocate enough space for string receival.

Sadly, MQL5 doesn’t have an ansistring type to handle conversions automatically. But, for the good part, at least in both cases strings are sent by reference, so it is actually a problem of meaning instead of a conflict in value/reference sending.

This means you have to send UNICODE strings that are to be correctly decoded as ANSI, and receive in ANSI strings that you have to convert to UNICODE for using.

When you are to receive an ANSI string in a UNICODE form, start reading UNICODE characters by typecasting each character to a unsigned short, then divide this in the two ANSI, (by modulo 256), add to resulting UNICODE string the modulo (as ANSI code), and the remainder (as ANSI code). So each 2 bytes of the original ANSI map into 4 bytes (2 UNICODE characters).

When you want to pack an ANSI-encoded UNICODE string, like an MQL5 string, as an ANSI, you read every two UNICODE chars in a row, then forcibly typecast them to unsigned char, like the size of ANSI characters. Then pack up new UNICODE character with the first read as modulo and second as remainder into a larger unsigned short, that you will add as code of the new character to the resulting UNICODE string.

The following is the code of two conversion functions, written as a include file. Make sure you make this file to be a include file, in the include folder, saving it as stringlib.mqh.

//+------------------------------------------------------------------+
//|                                                    stringlib.mqh |
//|                                       Copyright Bogdan Caramalac |
//|                                           http://mqlmagazine.com |
//+------------------------------------------------------------------+
#property library 
#property copyright "Bogdan Caramalac"
#property link      "http://mqlmagazine.com"
#property version   "1.00"
 
string ANSI2UNICODE(string s)
  {
   ushort mychar;
   long m,d;
   double mm,dd;
   string img;    
   string res="";
   if (StringLen(s)>0)
     {
      string g=" ";
      for (int i=0;i<StringLen(s);i++)
         {          
          string f="  ";          
          mychar=ushort(StringGetCharacter(s,i));
          mm=MathMod(mychar,256);
          img=DoubleToString(mm,0);
          m=StringToInteger(img);
          dd=(mychar-m)/256;
          img=DoubleToString(dd,0);
          d=StringToInteger(img);
          if (m!=0)
            {
             StringSetCharacter(f,0,ushort(m));
             StringSetCharacter(f,1,ushort(d));
             StringConcatenate(res,res,f);
            }//if (m!=0)
          else
            break;                      
         }//for (int i=0;i
      }//if (StringLen(s)>0)
   return(res);
  }
 
string UNICODE2ANSI(string s)
  {
   int leng,ipos;
   uchar m,d;
   ulong big;
   leng=StringLen(s);
   string unichar;
   string res="";
   if (leng!=0)
     {    
      unichar=" ";
      ipos=0;      
      while (ipos<leng)
        { //uchar typecasted because each double byte char is actually one byte
         m=uchar(StringGetCharacter(s,ipos));
         if (ipos+1<leng)
           d=uchar(StringGetCharacter(s,ipos+1));
         else
           d=0;
         big=d*256+m;                
         StringSetCharacter(unichar,0,ushort(big));         
         StringConcatenate(res,res,unichar);    
         ipos=ipos+2;
        }
     }
   return(res);
  }

When using the include you simply write

#include

as in the following example:

//+------------------------------------------------------------------+
//|                                                  teststrings.mq5 |
//|                                       Copyright Bogdan Caramalac |
//|                                           http://mqlmagazine.com |
//+------------------------------------------------------------------+
#property copyright "Bogdan Caramalac"
#property link      "http://mqlmagazine.com"
#property version   "1.00"
 
#include 
 
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
  {
   string original_unicode,ansi,converted_unicode;
   original_unicode="EvenString";
   ansi=UNICODE2ANSI(original_unicode);
   converted_unicode=ANSI2UNICODE(ansi);
   Print(original_unicode," -> ",ansi," -> ",converted_unicode);
   original_unicode="OddString";
   ansi=UNICODE2ANSI(original_unicode);
   converted_unicode=ANSI2UNICODE(ansi);
   Print(original_unicode," -> ",ansi," -> ",converted_unicode);
  }
//+------------------------------------------------------------------+