Certificate verify failed/expired error submitting job from nightly build
Getting a python exception when trying to submit job from latest modelbuilder CI.
- The first error message is
Exception [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1150)
. I'll paste the full dump in a details element below. - The error occurs when python tries to authenticate to the girder/cumulus server
https://ace3p.kitware.com
. That server uses a Let's Encrypt certificate, but I can access the site from a browser OK. - Looking further, I see that the problem occurs when using the superbuild python (which is 3.9.5) but things are OK when I use the python installed on my system (3.8.10).
- Viewing the certificate in the browser, it looks to me like it renewed last Thursday (Oct 7).
Click here for full console output
newt_sessionid 596aa04858de94f081e33a8bbdf90769
Exception [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1150)
Traceback (most recent call last):
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/lib/python3.9/site-packages/requests/packages/urllib3/connectionpool.py", line 557, in urlopen
httplib_response = self._make_request(conn, method, url,
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/lib/python3.9/site-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
self._validate_conn(conn)
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/lib/python3.9/site-packages/requests/packages/urllib3/connectionpool.py", line 784, in _validate_conn
conn.connect()
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/lib/python3.9/site-packages/requests/packages/urllib3/connection.py", line 247, in connect
self.sock = ssl_wrap_socket(conn, self.key_file, self.cert_file,
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/lib/python3.9/site-packages/requests/packages/urllib3/util/ssl_.py", line 305, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/bin/../lib/python3.9/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/bin/../lib/python3.9/ssl.py", line 1040, in _create
self.do_handshake()
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/bin/../lib/python3.9/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1150)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/lib/python3.9/site-packages/requests/adapters.py", line 366, in send
resp = conn.urlopen(
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/lib/python3.9/site-packages/requests/packages/urllib3/connectionpool.py", line 588, in urlopen
raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1150)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/share/cmb/workflows/ACE3P/internal/writers/nersc.py", line 77, in submit_ace3p
scope.cumulus = create_cumulus_client(scope, sim_item)
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/share/cmb/workflows/ACE3P/internal/writers/nersc.py", line 167, in create_cumulus_client
return CumulusClient(cumulus_host, scope.newt_sessionid)
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/share/cmb/workflows/ACE3P/internal/writers/cumulusclient.py", line 60, in __init__
r = self._session.put(url)
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/lib/python3.9/site-packages/requests/sessions.py", line 521, in put
return self.request('PUT', url, data=data, **kwargs)
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/lib/python3.9/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/lib/python3.9/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/home/john/temp/production/modelbuilder-ace3p-centos7-211011/lib/python3.9/site-packages/requests/adapters.py", line 447, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1150)
EXCEPTION: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1150)
LOG: INFO: smtk::operation::ImportPythonOperation: operation succeeded
DEBUG: Phrase handler: op ran 0x5256630
DEBUG: Phrase handler: op cre/pre 0x72e7d80
ERROR: Exception => [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1150)
INFO: smtk::operation::ImportPythonOperation: operation succeeded
ERROR: In /home/john/temp/production/modelbuilder-ace3p-centos7-211011/lib/python3.9/site-packages/smtk/io/__init__.py, line 54: Exception => [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1150)
ERROR: In /builds/gitlab-kitware-sciviz-ci/build/superbuild/smtk/src/smtk/operation/Operation.cxx, line 413: ACE3P.Export: operation failed
Edited by John Tourtellott