Trouble deploying SageMaker trained model in DeepLens

Possibly making some progress. Cloned incubator-mxnet to home directory and was able to run deploy.py on the model files with no error messages. Now when I run mo.optimize, I get the following output:

Python 2.7.12 (default, Dec  4 2017, 14:50:18) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mo
>>> mo.optimize('model_algo_1',540,540)
DEBUG:mo:DLDT command: python3 /opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/mo_mxnet.py --input_model /opt/awscam/artifacts/model_algo_1-0000.params --data_type FP16 --scale 1 --model_name model_algo_1 --output_dir /opt/awscam/artifacts --reverse_input_channels  --input_shape [1,3,540,540]
Model Optimizer arguments
        Batch:  1
        Precision of IR:        FP16
        Enable fusing:  True
        Enable gfusing:         True
        Names of input layers:  inherited from the model
        Path to the Input Model:        /opt/awscam/artifacts/model_algo_1-0000.params
        Input shapes:   [1,3,540,540]
        Log level:      ERROR
        Mean values:    ()
        IR output name:         model_algo_1
        Names of output layers:         inherited from the model
        Path for generated IR:  /opt/awscam/artifacts
        Reverse input channels:         True
        Scale factor:   1.0
        Scale values:   ()
        Version:        0.3.31.d8b314f6
        Prefix name for args.nd and argx.nd files: 
        Name of pretrained model which will be merged with .nd files: 
ERROR:mo:[ WARNING ]  
Detected not satisfied dependencies:
        mxnet: installed: 1.4.0, required: 1.0.0

Please install required versions of components or use install_prerequisites script
/opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/install_prerequisites/install_prerequisites_mxnet.sh
Note that install_prerequisites scripts may install additional components.
/usr/local/lib/python3.5/dist-packages/mxnet/module/base_module.py:56: UserWarning: You created Module with Module(..., label_names=['softmax_label']) but input with name 'softmax_label' is not found in symbol.list_arguments(). Did you mean one of:
        data
  warnings.warn(msg)
[ ERROR ]  No or multiple placeholders in the model, but only one shape is provided, can not set it.
[ ERROR ]  The following error happened while processing input shapes: . For more information please refer to Model Optimizer FAQ.

Further progress. Identified mismatch between model hyperparams and deploy.py inputs. Trained new model with resnet50 and image size 512, deployed, and ran deploy.py and mo.optimize.

>>> mo.optimize('model_algo_1',512,512)
DEBUG:mo:DLDT command: python3 /opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/mo_mxnet.py --input_model /opt/awscam/artifacts/model_algo_1-0000.params --data_type FP16 --scale 1 --model_name model_algo_1 --output_dir /opt/awscam/artifacts --reverse_input_channels  --input_shape [1,3,512,512]
Model Optimizer arguments
        Batch:  1
        Precision of IR:        FP16
        Enable fusing:  True
        Enable gfusing:         True
        Names of input layers:  inherited from the model
        Path to the Input Model:        /opt/awscam/artifacts/model_algo_1-0000.params
        Input shapes:   [1,3,512,512]
        Log level:      ERROR
        Mean values:    ()
        IR output name:         model_algo_1
        Names of output layers:         inherited from the model
        Path for generated IR:  /opt/awscam/artifacts
        Reverse input channels:         True
        Scale factor:   1.0
        Scale values:   ()
        Version:        0.3.31.d8b314f6
        Prefix name for args.nd and argx.nd files: 
        Name of pretrained model which will be merged with .nd files: 
ERROR:mo:[ WARNING ]  
Detected not satisfied dependencies:
        mxnet: installed: 1.4.0, required: 1.0.0

Please install required versions of components or use install_prerequisites script
/opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/install_prerequisites/install_prerequisites_mxnet.sh
Note that install_prerequisites scripts may install additional components.
/usr/local/lib/python3.5/dist-packages/mxnet/module/base_module.py:56: UserWarning: You created Module with Module(..., label_names=['softmax_label']) but input with name 'softmax_label' is not found in symbol.list_arguments(). Did you mean one of:
        data
  warnings.warn(msg)
[ ERROR ]  Cannot infer shapes or values for node "cls_prob".
[ ERROR ]  There is no registered "infer" function for node "cls_prob" with op = "softmax". Please implement this function in the extensions. For more information please refer to Model Optimizer FAQ.
[ ERROR ]  
[ ERROR ]  It can happen due to bug in custom shape infer function <UNKNOWN>.
[ ERROR ]  Or because the node inputs have incorrect values/shapes.
[ ERROR ]  Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ]  Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ]  Stopped shape/value propagation at "cls_prob" node. For more information please refer to Model Optimizer FAQ.

I've run the install_prerequisites script several times, but get the following error:

#details omitted
+ python3 pip3 install -r ../../../requirements_mxnet.txt
WARNING: The directory '/home/aws_cam/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: The directory '/home/aws_cam/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
ERROR: Could not open requirements file: [Errno 2] No such file or directory: '../../../requirements_mxnet.txt'

Any ideas?

Figured there might be some issues with my model, so I went back and ran thru the SageMaker example notebook for Object Detection, then deployed it. Initially got the same error as above, but then I rolled back MXNET to 1.3.0 for both PIP3 and PIP. Then when I ran deploy.rb and mo.optimize, I got the following result:

>>> mo.optimize('model_algo_1',512,512)
DEBUG:mo:DLDT command: python3 /opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/mo_mxnet.py --input_model /opt/awscam/artifacts/model_algo_1-0000.params --data_type FP16 --scale 1 --model_name model_algo_1 --output_dir /opt/awscam/artifacts --reverse_input_channels  --input_shape [1,3,512,512]
Model Optimizer arguments
        Batch:  1
        Precision of IR:        FP16
        Enable fusing:  True
        Enable gfusing:         True
        Names of input layers:  inherited from the model
        Path to the Input Model:        /opt/awscam/artifacts/model_algo_1-0000.params
        Input shapes:   [1,3,512,512]
        Log level:      ERROR
        Mean values:    ()
        IR output name:         model_algo_1
        Names of output layers:         inherited from the model
        Path for generated IR:  /opt/awscam/artifacts
        Reverse input channels:         True
        Scale factor:   1.0
        Scale values:   ()
        Version:        0.3.31.d8b314f6
        Prefix name for args.nd and argx.nd files: 
        Name of pretrained model which will be merged with .nd files: 
ERROR:mo:
(2, '')

According to the model optimizer documentation, a return of 2 for status means "Model optimization failed because you are using inconsistent platform versions." It recommends installing mxnet. Sigh.

I assume the "inconsistent platform versions" in mo.optimize status code 2 refers to the mxnet versions used to train and optimize the model. Is that correct?

As best as I can tell from Sagemaker documentation, the default mxnet version for training is 1.2.1.

I ran "sudo pip3 install mxnet==1.2.1" in deeplens, but still get error code 2.

Does anyone have more current or accurate info on mxnet versions in Sagemaker and DeepLens?

Is it possible the version conflict could be with deploy.py?

Edited by: BigEd on May 19, 2019 9:44 AM

Executed git reset command to set incubator-mxnet to the repo specified in the documentation. According to the README, that version of deploy.rb is compatible with mxnet 1.3.0 and earlier. Updated awscam version of mxnet to 1.3.0. Still getting return of status code 2 - inconsistent platforms from mo.optimize.

Believe I have resolved it. I updated awscam, rebooted, and installed mxnet version 1.4.0.

Documentation on how to use deploy.py is poor. For anyone else who is looking for the steps, do this:

1. SSH to your DeepLens or open a terminal on it, clone the incubator-mxnet/example/ssd repository to the home directory, then run the following line to roll the version back to the right one:

git reset --hard 73d88974f8bca1e68441606fb0787a2cd17eb364

2. Move files model_algo_1-0000.params, model_algo_1-symbol.json, and hyperparams.json from /opt/awscam/artifacts/ to /home/incubator-mxnet/example/ssd/.

3. Run deploy.py as follows:

sudo python deploy.py --network='resnet50' --epoch=0 --prefix='model_algo_1' --data-shape=512 --num-class=80

where shape is the x & y dimension of the images you trained the model on, num-class is the number of classes you trained it to identify, and network should match the network you trained to (VGG16 or resnet50). If you are successful, you should get this:

Saved model: model_algo_1-0000.params
Saved symbol: model_algo_1-symbol.json

4. Move the 3 model files back to /opt/awscam/artifacts/. You'll probably need to use sudo mv.

5. Launch sudo python, import mo, and run the optimizer.

Edited by: BigEd on May 23, 2019 8:17 PM

Edited by: BigEd on May 23, 2019 8:17 PM

Edited by: BigEd on May 23, 2019 8:19 PM