PermissionError: [Errno 13] Permission denied: 'alexnet-pretrained.pt'

[Camelliavin]

Hi, I'm trying End-To-End Confidential PyTorch Workflow.
When I run gramine-sgx ./pytorch pytorchexample.py, I got

Detected a huge manifest, preallocating 128MB of internal memory.
Gramine detected the following insecure configurations:

• loader.insecure__use_cmdline_argv = true (forwarding command-line args from untrusted host to the app)
• loader.insecure__use_host_env = true (forwarding environment vars from untrusted host to the app)
• sgx.allowed_files = [ ... ] (some files are passed through from untrusted host without verification)

Gramine will continue application execution, but this configuration must not be used in production!

Detected a huge manifest, preallocating 128MB of internal memory.
Traceback (most recent call last):
File "pytorchexample.py", line 8, in
alexnet = torch.load("alexnet-pretrained.pt")
File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 699, in load
with _open_file_like(f, 'rb') as opened_file:
File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 231, in _open_file_like
return _open_file(name_or_buffer, mode)
File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 212, in init
super(_open_file, self).init(open(name, mode))
PermissionError: [Errno 13] Permission denied: 'alexnet-pretrained.pt'
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 72, in apport_excepthook
from apport.fileutils import likely_packaged, get_recent_crashes
File "/usr/lib/python3/dist-packages/apport/init.py", line 5, in
from apport.report import Report
File "/usr/lib/python3/dist-packages/apport/report.py", line 32, in
import apport.fileutils
File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 27, in
from apport.packaging_impl import impl as packaging
File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in
import apt
File "/usr/lib/python3/dist-packages/apt/init.py", line 36, in
apt_pkg.init_system()
apt_pkg.Error: E:Error reading the CPU table

Original exception was:
Traceback (most recent call last):
File "pytorchexample.py", line 8, in
alexnet = torch.load("alexnet-pretrained.pt")
File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 699, in load
with _open_file_like(f, 'rb') as opened_file:
File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 231, in _open_file_like
return _open_file(name_or_buffer, mode)
File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 212, in init
super(_open_file, self).init(open(name, mode))
PermissionError: [Errno 13] Permission denied: 'alexnet-pretrained.pt'

Can someone help me?

[dimakuv]

@Camelliavin At which step exactly does it fail?

Did you do this step: https://gramine.readthedocs.io/en/latest/tutorials/pytorch/index.html#preparing-input-files

Did you do this step: https://gramine.readthedocs.io/en/latest/tutorials/pytorch/index.html#preparing-manifest-file

It's important to (a) download the file alexnet-pretrained.pt and to (b) mark it as Gramine-encrypted file ({ path = "/alexnet-pretrained.pt", uri = "file:alexnet-pretrained.pt", type = "encrypted" }).

Does the file alexnet-pretrained.pt exist on your machine? Maybe you accidentally deleted it or something. Can you show the output of ls -la alexnet-pretrained.pt? Can you show the output of head alexnet-pretrained.pt?

[jashan-iudx]

Facing the same issue. Could someone please help?

~/examples/pytorch-confidential$ ls -la alexnet-pretrained.pt
-rw-rw-r-- 1 jashan jashan 246964224 Apr  6 12:07 alexnet-pretrained.pt`

Error:

Traceback (most recent call last):
  File "pytorchexample.py", line 8, in <module>
    alexnet = torch.load("alexnet-pretrained.pt")
  File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 791, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 271, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 252, in __init__
    super().__init__(open(name, mode))
PermissionError: [Errno 13] Permission denied: 'alexnet-pretrained.pt'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 72, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 32, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 27, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 36, in <module>
    apt_pkg.init_system()
apt_pkg.Error: E:Error reading the CPU table

Original exception was:
Traceback (most recent call last):
  File "pytorchexample.py", line 8, in <module>
    alexnet = torch.load("alexnet-pretrained.pt")
  File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 791, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 271, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 252, in __init__
    super().__init__(open(name, mode))
PermissionError: [Errno 13] Permission denied: 'alexnet-pretrained.pt'

[dimakuv]

Did you encrypt alexnet-pretrained.pt after you downloaded it? In other words, did you run this exactly: https://gramine.readthedocs.io/en/latest/tutorials/pytorch/index.html#preparing-input-files

[jashan-iudx]

Yes

/examples/pytorch-confidential$ head alexnet-pretrained.pt 
����2�PF82�雠�r�0��<���'�FZ�(r�dJ>�-U��9�ȋ�q>h}�]�T 1��>���əK�{�VeRSrD����.�i��DB�݋��(��Hi� ﳇ����gM��
                                                                                           �����mb)�G���*G�s˱���_�D��Kke�|
�TՎ�������n�����%*�o{F6;�6m�m��F�n�b����P�8�?Av,븖�U��t�f�$&*�΁�p�6�t�"$�H(��}�K*8�#���I�T                                TiY���)^�לz"O)Q�_y�k�9'�O�����L�~�<�
                                                                                          aP�4�<(*���(�ڿ�l��/R� x���s�!�'�0���y��~�0q-�d]����ҾK�{/I�5i^�t����m��{�z�p��B}Dw��ێ�s8���ȏOI-0y�Ox3[���H$�������
                                                                                                                                                                                                           �Ti9TS�3��6���Nr=ĞX�;
                    �vs�wr̎H�N���X��I�cǶ��Q�jz�7�����U�j�!NBO���� 2�9_���9	�T@A���
�cY?'I�!��Fnk�9��e!�M���Ч���2;I�2�{h����.�iB�J��ϼڛ��~�>@ �鶱��b$�� ��%
                                                                      ?AjԽ��!�Ks%m��A���]��Τ9Z�G��̓J�]��A��!a"�S6�mƺ��Ԗ���*ҳ#�]-/��j⬉P��>��+�)&6��5�� ڄr9��P#i�(Abxn���g#������	\D�Z�w���$�*�e'\	�9y{��
N����Gq�6+
          r��9
�~��	���c�:S���ޓ�)��]��/4.�B�

This is my manifest template

# PyTorch manifest template

loader.entrypoint = "file:{{ gramine.libos }}"
libos.entrypoint = "{{ entrypoint }}"

loader.log_level = "{{ log_level }}"

loader.env.LD_LIBRARY_PATH = "/lib:/usr/lib:{{ arch_libdir }}:/usr/{{ arch_libdir }}"

# Restrict the maximum number of threads to prevent insufficient memory
# issue, observed on CentOS/RHEL.
loader.env.OMP_NUM_THREADS = "8"

loader.insecure__use_cmdline_argv = true
loader.insecure__use_host_env = true

loader.pal_internal_mem_size = "128M"

fs.mounts = [
  { uri = "file:{{ gramine.runtimedir() }}", path = "/lib" },
  { uri = "file:{{ arch_libdir }}", path = "{{ arch_libdir }}" },
  { uri = "file:/usr", path = "/usr" },
  { uri = "file:/etc", path = "/etc" },
  { uri = "file:{{ pillow_path }}", path = "{{ pillow_path }}" },

  { type = "tmpfs", path = "/tmp" },
  { path = "/classes.txt", uri = "file:classes.txt", type = "encrypted" },
  { path = "/input.jpg", uri = "file:input.jpg", type = "encrypted" },
  { path = "/alexnet-pretrained.pt", uri = "file:alexnet-pretrained.pt", type = "encrypted" },
  { path = "/result.txt", uri = "file:result.txt", type = "encrypted" },


]

# PyTorch loads its pre-trained models from here
# Add below uncommented line to fs.mounts array if you want to use torchvision.model.alexnet(pretrained=True)
# { type = "chroot", uri = "file:{{ env.HOME }}/.cache/torch", path = "{{ env.HOME }}/.cache/torch" }

sgx.enclave_size = "4G"
sgx.max_threads = 32
sgx.edmm_enable = {{ 'true' if env.get('EDMM', '0') == '1' else 'false' }}
sgx.nonpie_binary = true
sgx.remote_attestation = "dcap"

loader.env.LD_PRELOAD = "libsecret_prov_attest.so"
loader.env.SECRET_PROVISION_CONSTRUCTOR = "1"
loader.env.SECRET_PROVISION_SET_KEY = "default"
loader.env.SECRET_PROVISION_CA_CHAIN_PATH = "ssl/ca.crt"
loader.env.SECRET_PROVISION_SERVERS = "localhost:4433"

sgx.trusted_files = [
  "file:{{ gramine.libos }}",
  "file:{{ entrypoint }}",
  "file:{{ gramine.runtimedir() }}/",
  "file:{{ arch_libdir }}/",
  "file:/usr/{{ arch_libdir }}/",
  "file:{{ python.stdlib }}/",
  "file:{{ python.distlib }}/",
  "file:{{ pillow_path }}",
  "file:{{ python.get_path('stdlib', vars={'installed_base': '/usr/local'}) }}/",

  "file:pytorchexample.py",
"file:pytorchexample.py",
  "file:ssl/ca.crt"

  # Uncomment line below if you want to use torchvision.model.alexnet(pretrained=True)
  # "file:{{ env.HOME }}/.cache/torch/checkpoints/alexnet-owt-4df8aa71.pth",
]

sgx.allowed_files = [
]

# Gramine optionally provides patched OpenMP runtime library that runs faster inside SGX enclaves
# (add `-Dlibgomp=enabled` when configuring the build). Uncomment the line below to use the patched
# library. PyTorch's SGX perf overhead decreases on some workloads from 25% to 8% with this patched
# library. Note that we need to preload the library because PyTorch's distribution renames
# libgomp.so to smth like libgomp-7c85b1e2.so.1, so it's not just a matter of searching in the
# Gramine's Runtime path first, but a matter of intercepting OpenMP functions.
# loader.env.LD_PRELOAD = "/lib/libgomp.so.1"

[dimakuv]

Yes

How did you encrypt it? The binary form you copy-pasted doesn't look like proper Gramine FS encryption... (It should start with GRAFS in the first 5 bytes.)

[dimakuv]

Ah, I just checked, it's because head shows binary files in a weird way. Could you instead check with vim alexnet-pretrained.pt or less alexnet-pretrained.pt or some editor like this?

[jashan-iudx]

Sorry my bad I was using different wrap key for encryption and starting the server.