Failing to access Neptune endpoints using (aws)curl

0

Hi, I'm trying to query the SPARQL endpoint of Neptune v. 1.1.0.0. I am trying to use IAM auth with temp credentials (for a Neptune-specific IAM user) and I have Neptune configured with IAM db authentication: Enabled

This is my setup script that I run from EC2 (Ubuntu) that is in a public subnet in the same VPC as Neptune:

tmpfile=$(mktemp)

unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
unset AWS_SESSION_TOKEN

echo "AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID"
echo "AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY"
echo "AWS_SESSION_TOKEN: $AWS_SESSION_TOKEN"

aws sts assume-role \
        --profile default \
        --duration-seconds 1800 \
        --role-arn "arn:aws:iam::589606889449:role/NeptuneClient" \
        --role-session-name AWSCLI-Session > "$tmpfile"

AccessKeyId=$(cat "$tmpfile" | jq -r '.Credentials''.AccessKeyId')
SecretAccessKey=$(cat "$tmpfile" | jq -r '.Credentials''.SecretAccessKey')
SessionToken=$(cat "$tmpfile" | jq -r '.Credentials''.SessionToken')

export AWS_ACCESS_KEY_ID="$AccessKeyId"
export AWS_SECRET_ACCESS_KEY="$SecretAccessKey"
export AWS_SESSION_TOKEN="$SessionToken"

echo "AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID"
echo "AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY"
echo "AWS_SESSION_TOKEN: $AWS_SESSION_TOKEN"

aws sts get-caller-identity

get-caller-identity shows "Arn": "arn:aws:sts::589606889449:assumed-role/NeptuneClient/AWSCLI-Session" as expected.

The status endpoint works as well:

awscurl 'https://xxxxxxxxxxxxcluster.cluster-ro-cnol6sn9sq5j.us-east-1.neptune.amazonaws.com:8182/status' \
    --region us-east-1 \
    --service neptune-db

I get {"status":"healthy" ...} etc.

I was following the examples from Using RDF and SPARQL to access the graph in Amazon Neptune and replacing curl with awscurl. However if try to execute a query on a SPARQL endpoint, awscurl fails:

awscurl 'https://xxxxxxxxxxxxcluster.cluster-ro-cnol6sn9sq5j.us-east-1.neptune.amazonaws.com:8182/sparql' \
    -X POST \
    --data-binary \
    -d 'query=ASK {}' \
    --region us-east-1 \
    --service neptune-db

with this error:

Traceback (most recent call last):
  File "/home/ubuntu/.local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/awscurl.py", line 543, in main
    inner_main(sys.argv[1:])
  File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/awscurl.py", line 516, in inner_main
    response = make_request(args.request,
  File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/awscurl.py", line 107, in make_request
    canonical_request, payload_hash, signed_headers = task_1_create_a_canonical_request(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/awscurl.py", line 219, in task_1_create_a_canonical_request
    payload_hash = sha256_hash_for_binary_data(data) if data_binary else sha256_hash(data)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/utils.py", line 20, in sha256_hash_for_binary_data
    return hashlib.sha256(val).hexdigest()
TypeError: Strings must be encoded before hashing

If I remove the --data-binary arg

awscurl 'https://xxxxxxxxxxxxcluster.cluster-ro-cnol6sn9sq5j.us-east-1.neptune.amazonaws.com:8182/sparql' \
        -X POST \
        -d 'query=ASK {}' \
        --region us-east-1 \
        --service neptune-db

I get a Neptune error:

{"requestId":"2ec4e671-f734-5519-03fd-6971f5229051","detailedMessage":"Missing 'query' or 'update' parameter for POST request","code":"MissingParameterException"}

In normal curl the argument would be --data-binary 'query=ASK {}', but apparently not in awscurl because this this form fails too:

usage: awscurl [-h] [-v] [-i] [-X REQUEST] [-d DATA] [-H HEADER] [-k] [--data-binary] [--region REGION] [--profile PROFILE] [--service SERVICE] [--access_key ACCESS_KEY] [--secret_key SECRET_KEY] [--security_token SECURITY_TOKEN]
               [--session_token SESSION_TOKEN] [-L]
               uri
awscurl: error: unrecognized arguments:  query=ASK {}

Eventually I tried accessing the status endpoint using plain curl (version 7.81.0 (x86_64-pc-linux-gnu)) as described in this blog post:

curl 'https://xxxxxxxxxxxxcluster.cluster-ro-cnol6sn9sq5j.us-east-1.neptune.amazonaws.com:8182/status' -v \
    --aws-sigv4 "aws:amz:us-east-1:neptune-db" \
    --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
    --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" \
    --no-progress-meter

but this failed with yet another error:

{"requestId":"1ec4e65d-540b-93b3-a41f-9bd06f605385","detailedMessage":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.","code":"AccessDeniedException"}

This left me unsatisfied and exhausted :) Can someone please tell me what's going wrong here?

asked 9 months ago431 views
1 Answer
0

This worked!

awscurl 'https://xxxxxxxxxxxxcluster.cluster-ro-cnol6sn9sq5j.us-east-1.neptune.amazonaws.com:8182/sparql' \
        -X POST \
        -d 'query=ASK {}' \
        --region us-east-1 \
        --service neptune-db \
        --header 'Content-Type: application/x-www-form-urlencoded'

Setting --header is currently an awscurl workaround.

answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions