In the Julia REPL:
For using GPU, install & build:
]add CUDA ]build julia> using CUDA julia> using Transformers #run the model below . . .
Using pretrained Bert with
using Transformers using Transformers.Basic using Transformers.Pretrain ENV["DATADEPS_ALWAYS_ACCEPT"] = true bert_model, wordpiece, tokenizer = pretrain"bert-uncased_L-12_H-768_A-12" vocab = Vocabulary(wordpiece) text1 = "Peter Piper picked a peck of pickled peppers" |> tokenizer |> wordpiece text2 = "Fuzzy Wuzzy was a bear" |> tokenizer |> wordpiece text = ["[CLS]"; text1; "[SEP]"; text2; "[SEP]"] @assert text == [ "[CLS]", "peter", "piper", "picked", "a", "peck", "of", "pick", "##led", "peppers", "[SEP]", "fuzzy", "wu", "##zzy", "was", "a", "bear", "[SEP]" ] token_indices = vocab(text) segment_indices = [fill(1, length(text1)+2); fill(2, length(text2)+1)] sample = (tok = token_indices, segment = segment_indices) bert_embedding = sample |> bert_model.embed feature_tensors = bert_embedding |> bert_model.transformers
example folder for the complete example.
We have some support for the models from
using Transformers.HuggingFace # loading a model from huggingface model hub julia> model = hgf"bert-base-cased:forquestionanswering"; ┌ Warning: Transformers.HuggingFace.HGFBertForQuestionAnswering doesn't have field cls. └ @ Transformers.HuggingFace ~/peter/repo/gsoc2020/src/huggingface/models/models.jl:46 ┌ Warning: Some fields of Transformers.HuggingFace.HGFBertForQuestionAnswering aren't initialized with loaded state: qa_outputs └ @ Transformers.HuggingFace ~/peter/repo/gsoc2020/src/huggingface/models/models.jl:52
Current we only support a few model and the tokenizer part is not finished yet.
For more information
If you want to know more about this package, see the document and the series of blog posts I wrote for JSoC and GSoC. You can also tag me (@chengchingwen) on Julia's slack or discourse if you have any questions, or just create a new Issue on GitHub.
What we have before v0.2
TransformerDecodersupport for both 2d & 3d data.
Positionwisefor handling 2d & 3d input.
- docstring for most of the functions.
- runable examples (see
Transformers.HuggingFacefor handling pretrains from
What we will have in v0.2.0
- Complete tokenizer APIs
- more examples