Couple of questions on tsai

[strakehyr]

First of all, wanted to thank the author (@oguiza) for this amazing library. It's very solid, easy to use, and incorporates tons of functionality. I haven't found any other library out there that manages transformers for time-series data in such an easy way.

I also would like to ask some basic questions that have popped up while experimenting and trying the library out:

-I'll start with an easy question: is it normal for the transformers to overfit drastically? I have found that the models I have trained with the library overfit very quickly even with low learning rates and dropout incorporated. When using LSTMs I rarely saw any overfitting, with the same dataset. Further on that: the learning curves look nothing like with regular NNs. (using SGD), where I usually get an exponential curve. It sort of looks crazy to me:

The learning rate finder seems to work well, but probably a learning rate scheduler is very much needed to adjust for these swings.

• As mentioned in regression & forecasting fine-tuning #133 , is there a straightforward way to set up a multivariate multi-step forecast method? Be it with a recursive method (not really recommended in my experience) or by optimizing several steps? I have tried the solution mentioned in the link myself and have found that the model doesn't seem to recognize the different outputs as different timesteps but rather as different labels (which I think is not the same for the models understanding; correct me if I'm wrong).

• Are there any plans to incorporate recent literature developments in transformer structures such as the Infinite transformer (a.k.a. Informer) any time soon?

[vrodriguezf]

is it normal for the transformers to overfit drastically?

I don't know if it's normal, but defintely I've seen it too in tasks where I had few sequences (~ 200). Making the model smaller in terms of the number of layers, heads and the d_model helped more than dropout in my case.

is there a straightforward way to set up a multivariate multi-step forecast method?

As long as your y has the correct shape, i.e., bs x n_vars x seq_len and you are using a model ending in Plus, you should be able to do it...not sure I understand what your problem exactly is.

[oguiza]

is it normal for the transformers to overfit drastically?
Transformers are known for requiring a large number of samples. So to answer your question, yes they may overfit. Having said that, it doesn't seem based on your charts that you are overfitting the train set. The loss still seems pretty high. It seems the model is not training well. I've seen that when data wasn't correctly scaled. Are you using any type of scaling (TSStandardize, TSNormalize, etc)?
is there a straightforward way to set up a multivariate multi-step forecast method?
Yes. If the y is 3d [samples x #vars x steps] the output of the model should also be 3d, with the same number of variables and steps as the target. It uses a direct method (not a recursive one), creating all steps at the same time.
Are there any plans to incorporate recent literature developments in transformer structures such as the Infinite transformer (a.k.a. Informer) any time soon?
I'm planning to integrate them but not soon.

[strakehyr]

I am using TSStandardize. I am using it to forecast 24 hours of a single variable based on other 25 variables.

Full code (as seen in the issues post, and corrected with reduced parameters and a bit of dropout as recommended per @vrodriguezf ) here:

train_ratio=0.7

look_back = 24

horizon=24

df = pd.read_csv(file_path)

train_end = round(len(df)*train_ratio)

Time_index = df['Time']

df.drop('Time', inplace=True, axis=1)

X, y = SlidingWindow(look_back, get_x = list(df.iloc[:,1:].columns), get_y = (df.iloc[:,0].name), horizon=horizon)(df)

y = y.reshape(y.shape[0], 1, y.shape[1])

splits = get_splits(y, n_splits=1, valid_size=.2, shuffle=True)

check_data(X, y, splits)

tfms  = [None, [TSRegression()]]

batch_tfms = TSStandardize(by_sample=True, by_var=True)

dls = get_ts_dls(X, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms, bs=24)

learn = ts_learner(dls, TSTPlus, fc_dropout = .5, ks = 5, n_layers = 1, d_model= 32, n_heads = 4, opt_func = SGD, loss_func = mse, metrics=[mae, mse], cbs=ShowGraph())


lr = learn.lr_find()[0]

learn.loss_func

learn.fit_one_cycle(10, lr)

While the model seems to be learning better it still seems a bit unsteady and the learning curve doesn't show much resemblance with the classic exponential function.

If you have any further tips (maybe you propose some other model or some other data preparation for my case) I would be very thankful to hear them.

[strakehyr]

Also wanted to update on the fact that the models sometimes output nan metrics when training (Even though there are no nan values in either X or y):

learn.fit_one_cycle(30, lr)
█epoch     train_loss  valid_loss  mae       mse       time    
██0         1036.523071 1246.596924 22.063200 1246.59692400:12     
██1         1113.060425 1241.295410 22.097742 1241.29541000:12     
██2         1053.889038 1246.754272 21.785574 1246.75427200:12     
██3         1157.709717 1249.881958 23.172104 1249.88195800:12     
██4         1157.555420 1223.927856 21.412077 1223.92785600:12     
██5         1174.137207 1220.956177 22.542295 1220.95617700:12     
██6         nan         nan         nan       nan       00:12     
██7         nan         nan         nan       nan       00:12     
██8         nan         nan         nan       nan       00:12     
██9         nan         nan         nan       nan       00:12     
██10        nan         nan         nan       nan       00:12     
██11        nan         nan         nan       nan       00:12  

[oguiza]

Hi @strakehyr,
I'd suggest that you start using the default settings. After some initial tests, you may then need to modify a few.

You can start with something like this:

tfms  = [None, TSRegression()]
batch_tfms = TSStandardize(by_var=True)
dls = get_ts_dls(X, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms)
learn = ts_learner(dls, TSTPlus, metrics=[mae, mse], cbs=ShowGraph())

I don't know anything about the data, so it's a bit difficult to know where the nan values are coming from.
Something I always do is to try to achieve the desired performance on the training set. This will help you determine the number of epochs required. It may be 5 or 500. You'll only know when you try to overfit the training set.

[strakehyr]

Hi @oguiza,
just tried it and it worked wonders! I probably think this might be due to the by_sample argument in the dls (either that or the batch size). Thank you fondly, and mind the inconvenience.

[oguiza]

Great! I'm happy to hear that.