-
Notifications
You must be signed in to change notification settings - Fork 33.4k
[Time-Series] informer model #21099
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
[Time-Series] informer model #21099
Changes from all commits
Commits
Show all changes
181 commits
Select commit
Hold shift + click to select a range
3a3a481
added informer to gitignore
elisim 9f8e00d
added informer to gitignore
elisim 965eeb6
WIP informer2020
elisim a0231bf
Merge branch 'add-informer2020-model' of https://github.com/elisim/tr…
elisim 1fcecdc
Merge remote-tracking branch 'upstream/main' into add-informer2020-model
elisim ec34ab1
added checking that instantiate works
elisim f274b76
added config using gluonTS by kashif
elisim b35a8b2
WIP config
elisim 27ebb73
adding informeConfig. need to remove FeatureEmbedder
elisim 9d4ae99
done InformerConfig, but need to change the names
elisim 1223ba5
Done informer model init. working on enc-dec
elisim a7d38dc
added things to address, after reading again enc-dec in the paper
elisim 8853e93
done modeling - checking initialization work
elisim 422cdf5
added informer to gitignore
elisim c1001f0
WIP informer2020
elisim 8648e08
added checking that instantiate works
elisim 5bfe844
added config using gluonTS by kashif
elisim b29cfa6
WIP config
elisim 215b030
adding informeConfig. need to remove FeatureEmbedder
elisim d56a0cd
done InformerConfig, but need to change the names
elisim f68192d
Done informer model init. working on enc-dec
elisim ffb7833
added things to address, after reading again enc-dec in the paper
elisim 4286dd3
done modeling - checking initialization work
elisim 622a921
Merge branch 'add-informer2020-model' of https://github.com/elisim/tr…
elisim d278b80
moved enc-dec init to InformerEncoder/Decoder init
elisim 9e66890
added 'init_std' to config, now model init works!
elisim d9d1a67
WIP conversion script, and added code sources
elisim eedd4ca
Merge remote-tracking branch 'upstream/main' into add-informer2020-model
elisim 52de756
WIP conversion script: loading original informer pth works
elisim cb9a287
WIP conversion script: change defaults in the config
elisim 2b8d7e4
WIP conversion script: supporting Informer input embedding
elisim 3fa969a
WIP conversion script: added parameters for the informer embed
elisim 49170d8
WIP conversion script: change dim_feedforward=2048
elisim 585e40b
WIP conversion script: remove unused args for loading checkpoint
elisim 824f48e
just cleaning up
elisim 878bd79
DataEmbedding removed, after thinking with Kashif
elisim 60ee685
Merge remote-tracking branch 'upstream/main' into add-informer2020-model
elisim 1ce4310
working on forward pass
elisim 4568c0c
WIP forward pass: trying to establish working batch for forward pass
elisim a31fc5a
cleaning and finalizing
elisim 27045b8
adding HF names and docs
elisim c8a6cae
init after cleaning works
elisim 0e8ffb6
WIP in tests
elisim 83d39df
added docs for the informer specific args
elisim fdffeb8
fix style
kashif aab1e08
undo change
kashif f8a33c5
Merge branch 'huggingface:main' into add-informer2020-model
kashif 4e73184
cleaning informer, now need to work only enc-dec
elisim f12c0b0
initial enc-dec classes
kashif 48337b0
added encoder and decoder
kashif 5374049
added todo
kashif dd3b46a
add todos for conv_layers
kashif 6bf419e
added decoder docs from vanilla
elisim ed21221
added encoder docs from vanilla
elisim e4c7875
remove encoder decoder from the original informer
elisim 645810a
removed AttentionLayer from the original paper
elisim 2529b7c
removed TriangularCausalMask, same as decoder_attention_mask
elisim 225af56
initial sparse attention
kashif f31ec98
use conv_layers
kashif 778c2a2
fixed test_config test
elisim 7363444
fix parenthesis when itearting zip(layers, conv_layers)
elisim 86486d2
error found in prob attention, added sizes as comments
elisim 606ca81
fix sizes
kashif b416de8
added proposal for q_reduce indexing, and remove unused
elisim 3bfbb23
WIP ProbMask, and changed factor=2 for testing
elisim 3ebec34
Merge remote-tracking branch 'upstream/main' into add-informer2020-model
elisim 67e1964
Merge remote-tracking branch 'upstream/main' into add-informer2020-model
elisim 11a081e
remove unused libs for this PR for creating the env
elisim efdbff6
Merge remote-tracking branch 'upstream/main' into add-informer2020-model
elisim bb70b1b
fix checking the attn_weights.size() after bmm
elisim 588205b
Q_reduce: changed from torch.gather to simple slicing
elisim b3f595f
WIP calculate final attn_output
elisim ceef0ad
finish adding v_aggregated, attn_output ready
elisim db95fd8
changed tgt_len to u in attention_mask, need to fix the size error
elisim 3946b0c
comment attention_mask for encoder, and fix if cond for v_agg
elisim d2db148
added ProbMask support (wip), removed old original code
elisim 4477d42
finished ProbMask 😃
elisim fa131dc
Revert "remove unused libs for this PR for creating the env"
elisim 19a3408
fixes
kashif 84a4ba3
make style
kashif a48e661
fix initial tests
kashif a3b6ddb
fix more tests
kashif a141712
dry
kashif 299d17f
make style
kashif 36df283
remove unused files
kashif 0b4220a
style
kashif e9e3c6e
added integration tests
kashif 83657df
fix num_static_real_features
kashif c893ad8
fix header
kashif 79a21e0
remove unused function
kashif db1a0e4
fix example
kashif ad96fdd
fix docs
kashif ea921d9
Update src/transformers/models/informer/configuration_informer.py
kashif 6ae25e6
Update src/transformers/models/informer/modeling_informer.py
kashif fd046ba
Update src/transformers/models/informer/configuration_informer.py
kashif 18751ae
Update src/transformers/models/informer/configuration_informer.py
kashif 51f890d
Update src/transformers/models/informer/configuration_informer.py
kashif dc49ff7
Update src/transformers/models/informer/configuration_informer.py
kashif 125ee05
fixes for reviewer
kashif a64162a
use prediction_length from model
kashif 19e3279
fix style
kashif b316a5a
fixed informer.mdx
kashif 35e4549
added to index
kashif 31f8bd3
updated readme
kashif c26ba0e
undo
kashif 8379d0c
make fix-copies
kashif a8846f5
typo
kashif 217260d
fix copy
kashif 1421ccd
added Informer to toctree
kashif 2b0a26d
in order
kashif 10ed995
fixed comments
kashif ea07cd9
remove unneeded new lines in docs
kashif 8299745
make static real and cat optional
kashif 4866362
fix use of distil conv layers
kashif 8218d55
fixed integration test
kashif 21e3d43
added checkpoint for convlayer
kashif a0204e0
Merge branch 'main' into add-informer2020-model
kashif c1a58eb
make fix-copies
kashif 56578bf
updated from time series model
kashif 32e94d1
make fix-copies
kashif 29ba424
copy decoder
kashif 19abd0b
fix unit tests
kashif 39ce9fd
updated scaling config
kashif 68e68c0
fix integration tests
kashif 8ee1b10
IGNORE_NON_TESTED
kashif 05856aa
IGNORE_NON_AUTO_CONFIGURED
kashif 50fc144
IGNORE_NON_AUTO_CONFIGURED
kashif e7845e8
updated check configs
kashif b030bea
fix formatting
kashif 308268b
undo change from time series
kashif f064080
prediction_length should not be None
kashif 5086c4c
aliign with the blog: prettify ProbSparse and change attention_factor…
elisim d47c556
make style
elisim e08566f
Merge remote-tracking branch 'upstream/main' into add-informer2020-model
kashif 5462fbd
make fix-copies
kashif ae22d55
Merge remote-tracking branch 'upstream/main' into add-informer2020-model
elisim 4988b2f
niels CR: update contributed by
elisim c7be3cf
niels CR: update configuration_informer.py
elisim bb03fc6
niels CR: update kashif -> huggingface
elisim 1e78712
niels CR: `sampling_factor` only relevant when `attention_type`=prob
elisim 4d3ec7a
Merge branch 'add-informer2020-model' of https://github.com/elisim/tr…
elisim 2afa495
make style
elisim 193938f
fixed U_part: added multiplication by `L_Q`
elisim f064beb
fixed bug: remove `is not None` from `if config.distil`
elisim 3db65bd
fixed test: `decoder_seq_length` to `encoder_seq_length` in cross_att…
elisim c4c6133
fix integration tests
kashif 329647e
updated model hub
kashif 582c867
do not shift as in training
kashif 25867e2
undo
kashif 7e8420e
fix make-copies
kashif aeff495
Merge remote-tracking branch 'upstream/main' into add-informer2020-model
kashif 3074c73
make fix-copies
kashif e455bbf
Merge remote-tracking branch 'upstream/main' into add-informer2020-model
elisim b4cbddf
added `if prediction_length is None`
elisim b934cb8
changed `ProbSparseAttention` to `InformerProbSparseAttention`
elisim 426d14f
changed `V_sum` -> `v_mean_dim_time`
elisim 73d1840
changed `ConvLayer` to `InformerConvLayer` and fixed `super()`
elisim 3bc6163
TimeSeriesTansformer->Informer in decoder's Copied from
elisim ba6a8c3
more descriptive in ProbSparse
elisim f4fde7a
make style
elisim 65811d1
fix coped from
kashif 18fd0f8
Revert "added `if prediction_length is None`"
kashif f9e3bee
fixed indent
kashif e13dfab
use InformerSinusoidalPositionalEmbedding
kashif beba0a0
Merge branch 'main' of https://github.com/huggingface/transformers in…
kashif b873c38
make fix-style
kashif f557964
fix from #21860
kashif d58f3ab
fix name
kashif e1b746a
Merge branch 'main' of https://github.com/huggingface/transformers in…
kashif 3bf819d
make fix-copies
kashif 3703738
use time series utils
kashif 240ac6a
fix dec num_heads
kashif c638f24
docstring
kashif 9608217
added time series util doc
kashif 0bc09de
_import_structure
kashif bd724fc
formatting
kashif db6e0bd
changes from review
kashif b14aa34
make style
kashif e36f6c0
fix docs
kashif 0ac82bb
fix doc
kashif e5eff8a
removed NegativeLogLikelihood
kashif File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.