Difference in pytorch BERT model output vs neuron

Question

I converted a pytorch BERT model to neuron. However the embedding or output tensors which is a list of 1024 size is different..i.e the list sizes are same but individual entries differ. Each of the numbers differs around 1-5% with the original pytorch model output. 
This is the code that I use to neuron compile the pytorch model.
``` 
from transformers import BertTokenizer, BertModel
import torch
import torch_neuron
import os.path
import os
import numpy

tokenizer = BertTokenizer.from_pretrained(modelname, model_max_length=512)
input_str = "The patient's ability is determined based on patients medical parameters, patients history of ability to attend a remote clinician sessions, and physical parameters. Based on identified parameters a patients profile score is calculated to determine patients ability to attend the remote clinician session."
inputs = tokenizer(input_str, padding='max_length', return_tensors="pt")
PATH = './ptmodel/'
fname = 'modelneuron.pt'
kwargs = {'compiler_args':['--fast-math', 'none','--neuroncore-pipeline-cores', '1']}
model = BertModel.from_pretrained(PATH, local_files_only=True, return_dict=False)
neuron_model = torch_neuron.trace(model,
                                    example_inputs = (inputs['input_ids'],inputs['attention_mask'],inputs['token_type_ids']), **kwargs)
neuron_model.save(fname)
```
Has anyone faced this issue or knows how to solve this??

Thanks
Ajay

Answer

Sample-to-sample variation is expected since CPU architecture is different from Inferentia (and different from GPU), and the order of summation can lead to slightly different results. Will you be able to measure the accuracy over the evaluation data set for both CPU and Inferentia (and GPU also if it is available)?

Answer

The "--fast-math=none" option you are using is actually internally casting matrix multiply operations to fp16 and is maybe the best option for you already. Some precision loss is expected due to the use of lower precision datatype. However, over a dataset such as MRPC, we see the same BERT accuracy as GPU/CPU. If you see otherwise, please file a ticket via https://github.com/aws-neuron/aws-neuron-sdk/issues or send an email to aws-neuron-support@amazon.com .

Difference in pytorch BERT model output vs neuron

Relevant content