By using AWS re:Post, you agree to the Terms of Use
/Amazon Neptune/

Questions tagged with Amazon Neptune

Sort by most recent
  • 1
  • 90 / page

Browse through the questions and answers listed below or filter and sort to narrow down your results.

Unable to perform OpenSearch text queries from Gremlin using AWS Lambda written in Javascript

I am syncing my AWS Neptune nodes in an AWS OpenSearch cluster as per the documentation https://docs.aws.amazon.com/neptune/latest/userguide/full-text-search.html. The name of the OpenSearch index is amazon_neptune. The OpenSearch index type is _doc. Following is the index configuration ``` { "settings": { "number_of_shards": 1, "number_of_replicas": 1, "analysis": { "normalizer": { "useLowercase": { "type": "custom", "filter": "lowercase" } } } }, "mappings": { "properties": { "document_type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "entity_id" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "entity_type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "predicates": { "properties": { "content": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above" : 1000, "normalizer": "useLowercase" } } }, "visibilityType": { "type": "keyword" }, "status": { "type": "keyword" }, "type": { "type": "keyword" }, "firstName": { "type": "text", "fields": { "keyword": { "type": "keyword", "normalizer": "useLowercase" } } }, "lastName": { "type": "text", "fields": { "keyword": { "type": "keyword", "normalizer": "useLowercase", "ignore_above" : 1000 } } } } } } } } ``` Using the npm gremlin package, I'm trying to query my documents. Following is the code: ``` 'use strict'; const gremlin = require('gremlin'); exports.handler = async (event, context) => { try { const DriverRemoteConnection = gremlin.driver.DriverRemoteConnection; const Graph = gremlin.structure.Graph; const dc = new DriverRemoteConnection(<neptune_endpoint>,{}); const graph = new Graph(); const dbClient = graph.traversal().withRemote(dc); const res = await dbClient .withSideEffect("Neptune#fts.endpoint",<https_opensearch_endpoint>) .withSideEffect('Neptune#fts.queryType', 'term') .V().has("visibilityType","Neptune#fts PUBLIC") .toList(); console.log('res:', res); } catch(err) { console.error('Failed to query', err); } } ``` But I'm getting the following error ``` Failed to query ResponseError: Server error: {"detailedMessage":"method [POST], host [<https_opensearch_endpoint>], URI [/amazon_neptune/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 403 Forbidden]\n{\"Message\":\"User: anonymous is not authorized to perform: es:ESHttpPost\"}","requestId":"23a9e7d7-7dde-465b-bf29-9c59cff12e86","code":"BadRequestException"} (500) ``` I have given the following permission to my lambda ``` Type: AWS::IAM::Policy Properties: PolicyName: <Policy_Name> Roles: - 'Ref': <lambda_role> PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: - es:ESHttpGet - es:ESHttpPost - es:ESHttpPut - es:ESHttpDelete Resource: <opensearch_cluster_arn> ``` My OpenSearch cluster as well as Neptune cluster are located inside the same VPC. My lambda is hosted inside the same VPC as well. Please help me in understanding why I'm getting the 403 error when I've given the proper reading permissions to my lambda. Any help would be highly appreciated.
2
answers
0
votes
117
views
asked a month ago

Storing/representing a hierarchical tree used for navigation within an Amplify web app and AppSync GraphQL API layer.

Hi, **TL;DR: Can anyone give a recommend approach to storing customisable n-level hierarchy trees for grouping and navigating results via a frontend Amplify-powered web app (using DynamoDB or any other database solution that can be mapped to AppSync ideally)?** **Some background** I'm building an multi-tenant IoT analytics solution that takes data from some sensors out in the field, uploads to AWS, processes this data and stores in a DynamoDB table (i.e. a very "standard" setup). I'm planning on adding a web frontend (built using Amplify and an AppSync GraphQL layer) that will allow users to navigate a **customisable, n-level** hierarchy tree of assets, in order to view the sensor data we've collected. Examples of valid hierarchies include: Country -> Site -> Building -> Floor -> Room -> Sensor (6-level) or Site -> Building -> Room -> Sensor (4-level) etc. The important thing here, is that this hierarchy tree can differ per customer, and needs to be customisable on a tenant-by-tenant basis, but we don't need to do any complex analysis or navigation of relationships between hierarchy levels (so, to me, something like Amazon Neptune or another graph database feels a bit overkill, but perhaps I’m wrong). My first thought was to try and build a hierarchical relationship inside of a DynamoDB table, possibly making use of a GSI to provide this, but in all of the examples I’ve seen online, the focus is very much on quick retrieval, but not so quick updating of hierarchy trees – now, whilst it’s unlikely that these tree structures would be updated on a regular basis, it is something we need to be able to support, so the idea of possibly updating ‘000s of rows in DynamoDB every time we want to make a change to the hierarchy tree for a given control area doesn’t seem quite right to me. Hence, my question above. I'm ideally looking for guidance on how to structure a DDB table to best support BOTH optimal retrieval of, and updates to, hierarchy trees in our application, but if DDB isn't the right answer here, then suggestions of alternatives would also be greatly appreciated. Many thanks in advance.
1
answers
0
votes
23
views
asked 2 months ago

Unable to update Neptune cluster to engine version 1.1.0.0

I am trying to update my Neptune cluster from engine version 1.0.5.1.R2 to the latest 1.1.0.0 version, but am unable to do that either via the console (only 1.0.5.1.R2 is in drop down), or using the AWS CLI as suggested in the [release notes](https://docs.aws.amazon.com/neptune/latest/userguide/engine-releases-1.1.0.0.html). The CLI command I am using is like follows: ``` aws neptune modify-db-cluster \ --db-cluster-identifier my-neptune-cluster \ --engine neptune \ --engine-version 1.1.0.0 \ --allow-major-version-upgrade \ --apply-immediately \ --db-cluster-parameter-group-name my-neptune-pg \ --db-instance-parameter-group-name my-neptune-pg ``` This fails with the error: ``` An error occurred (InvalidParameterCombination) when calling the ModifyDBCluster operation: Cannot upgrade neptune from 1.0.5.1 to 1.1.0.0 ``` Here are the parameters that are set on both the cluster and database: ``` $ aws neptune describe-db-cluster-parameters --db-cluster-parameter-group-name my-neptune-pg | jq -r '.Parameters[] | [.ParameterName, .ParameterValue] | @csv' "neptune_autoscaling_config","" "neptune_enable_audit_log","0" "neptune_enforce_ssl","0" "neptune_lab_mode", "neptune_lookup_cache","1" "neptune_ml_endpoint", "neptune_ml_iam_role", "neptune_query_timeout","120000" "neptune_streams","0" $ aws neptune describe-db-parameters --db-parameter-group-name my-neptune-pg | jq -r '.Parameters[] | [.ParameterName, .ParameterValue] | @csv' "neptune_query_timeout","120000" "neptune_result_cache","0" ```
1
answers
0
votes
31
views
asked 6 months ago

Big gremlin insert query (sometimes) results with InternalFailureException

Hi, we are sometimes experiencing issues with big insert queries. We are running Neptune Engine 1.1.0.0 on a db.r5.large instance. Our initial approach is to insert new data which includes masterdata also (possibly new). To ensure the masterdata will not be created twice, we use appropriate coalesce-steps. A resulting query includes ~ 8000 gremlin steps for example. This query takes more than 20 seconds to execute on an empty database. Our second approach was to assume that all master data already existed in the database. Therefore no more coalesce statement is needed to insert our new data. The corresponding query of our first attempt reduces then to ~ 5000 gremlin steps. This query fails with an InternalFailureException. A descriptive reason is not given. To analyze why the query of our second approach fails, we cut it in half which first also fails (InternalFailureException). Next we tried the quarter of the query, which then succeeded. We retried to run again half of the query which than also succeeded. After that we even were able to run the full query. Please notice, that after each try we reset the database to only include our masterdata. We are wondering why our insert statement runs over 20 seconds on an empty database. Are we doing something wrong here? On the other hand we are wondering why our second approach sometimes work and sometimes fails with the InternalFailureException. It seems that this behavior is undeterministic. What can be the reason for this? How can we fix the query, so that it works everytime we retry on our reset database, including our masterdata? You can access the queries here: 1. Initial, full query with COALESCE steps: https://pastebin.com/d4u6edTH 2. Query to drop all data except master data (to execute after execution of query 1): https://pastebin.com/z8iLeumM 3. Our second approach query without COALESCE: https://pastebin.com/71WCC1iM
1
answers
0
votes
20
views
asked 6 months ago

Constructor parameter should be str in Notebook

I have a Python 3 Neptune Notebook. When I try to run a command such as result = g.V().has('code','test').valueMap().toList() print('result is ' + str(result)) I get: **~/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/yarl/_url.py in __new__(cls, val, encoded, strict)** **156 val = urlsplit(str(val))** **157 else:** **--> 158 raise TypeError("Constructor parameter should be str")** **159** **160 if not encoded:** **TypeError: Constructor parameter should be str** This is a new error, these commands worked for the past year or so. This is a similar error I get in python when I use Python 3.8 instead of 3.7 to call Neptune... is it possible for me to change the notebook Python to 3.7? Edited by: ColinWithMothrazzz on Nov 18, 2021 6:09 AM Over the past week or so, in order to run a notebook I have been downgrading my notebook using this command: %%bash aws s3 cp s3://aws-neptune-notebook/graph_notebook.tar.gz /tmp/graph_notebook.tar.gz rm -rf /tmp/graph_notebook tar -zxvf /tmp/graph_notebook.tar.gz -C /tmp /tmp/graph_notebook/install.sh -v="3.0.6" to run the code. Now I am unable to downgrade the notebook, when I try I get this: **WARNING: Skipping NeptuneGraphNotebook as it is not installed.** **Config option `kernel_spec_manager_class` not recognized by `InstallNBExtensionApp`.** **Installing /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/gremlin_syntax/static -> gremlin_syntax** **Removing: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/gremlin_syntax** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/gremlin_syntax/** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/gremlin_syntax/static/__init__.py -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/gremlin_syntax/__init__.py** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/gremlin_syntax/static/main.js -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/gremlin_syntax/main.js** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/gremlin_syntax/static/description.yaml -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/gremlin_syntax/description.yaml** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/gremlin_syntax/static/README.md -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/gremlin_syntax/README.md** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/gremlin_syntax/__pycache__** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/gremlin_syntax/static/__pycache__/__init__.cpython-36.pyc -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/gremlin_syntax/__pycache__/__init__.cpython-36.pyc** **- Validating: OK** **Installing /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/sparql_syntax/static -> sparql_syntax** **Removing: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/sparql_syntax** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/sparql_syntax/** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/sparql_syntax/static/__init__.py -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/sparql_syntax/__init__.py** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/sparql_syntax/static/main.js -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/sparql_syntax/main.js** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/sparql_syntax/static/description.yaml -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/sparql_syntax/description.yaml** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/sparql_syntax/static/README.md -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/sparql_syntax/README.md** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/sparql_syntax/__pycache__** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/sparql_syntax/static/__pycache__/__init__.cpython-36.pyc -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/sparql_syntax/__pycache__/__init__.cpython-36.pyc** **- Validating: OK** **Installing /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/opencypher_syntax/static -> opencypher_syntax** **Removing: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/opencypher_syntax** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/opencypher_syntax/** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/opencypher_syntax/static/__init__.py -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/opencypher_syntax/__init__.py** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/opencypher_syntax/static/main.js -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/opencypher_syntax/main.js** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/opencypher_syntax/static/description.yaml -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/opencypher_syntax/description.yaml** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/opencypher_syntax/static/README.md -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/opencypher_syntax/README.md** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/opencypher_syntax/__pycache__** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/opencypher_syntax/static/__pycache__/__init__.cpython-36.pyc -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/opencypher_syntax/__pycache__/__init__.cpython-36.pyc** **- Validating: OK** **Installing /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/neptune_menu/static -> neptune_menu** **Removing: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/neptune_menu** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/neptune_menu/** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/neptune_menu/static/__init__.py -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/neptune_menu/__init__.py** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/neptune_menu/static/main.js -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/neptune_menu/main.js** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/neptune_menu/static/description.yaml -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/neptune_menu/description.yaml** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/neptune_menu/static/README.md -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/neptune_menu/README.md** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/neptune_menu/__pycache__** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/neptune_menu/static/__pycache__/__init__.cpython-36.pyc -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/neptune_menu/__pycache__/__init__.cpython-36.pyc** **- Validating: OK** **Installing /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/playable_cells/static -> playable_cells** **Removing: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells/** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/playable_cells/static/playable_cells.css -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells/playable_cells.css** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/playable_cells/static/__init__.py -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells/__init__.py** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/playable_cells/static/main.js -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells/main.js** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/playable_cells/static/description.yaml -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells/description.yaml** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/playable_cells/static/README.md -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells/README.md** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells/__pycache__** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/playable_cells/static/__pycache__/__init__.cpython-36.pyc -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells/__pycache__/__init__.cpython-36.pyc** **- Validating: OK** **To initialize this nbextension in the browser every time the notebook (or other app) loads:** **jupyter nbextension enable graph_notebook.nbextensions --py --sys-prefix** **Config option `kernel_spec_manager_class` not recognized by `EnableNBExtensionApp`.** **Enabling notebook extension gremlin_syntax/main...** **- Validating: OK** **Enabling notebook extension sparql_syntax/main...** **- Validating: OK** **Enabling notebook extension opencypher_syntax/main...** **- Validating: OK** **Enabling notebook extension neptune_menu/main...** **- Validating: OK** **Enabling notebook extension playable_cells/main...** **- Validating: OK** **Config option `kernel_spec_manager_class` not recognized by `InstallNBExtensionApp`.** **Installing /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/widgets/nbextension/static -> graph_notebook_widgets** **Out of date: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/graph_notebook_widgets/__init__.py** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/widgets/nbextension/static/__init__.py -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/graph_notebook_widgets/__init__.py** **Up to date: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/graph_notebook_widgets/extension.js** **Up to date: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/graph_notebook_widgets/index.js** **Out of date: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/graph_notebook_widgets/__pycache__/__init__.cpython-36.pyc** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/widgets/nbextension/static/__pycache__/__init__.cpython-36.pyc -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/graph_notebook_widgets/__pycache__/__init__.cpython-36.pyc** **- Validating: OK** **To initialize this nbextension in the browser every time the notebook (or other app) loads:** **jupyter nbextension enable graph_notebook.widgets --py --sys-prefix** **Config option `kernel_spec_manager_class` not recognized by `EnableNBExtensionApp`.** **Enabling notebook extension graph_notebook_widgets/extension...** **- Validating: OK** **/home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin/python: No module named graph_notebook.start_notebook** **DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.** Edited by: ColinWithMothrazzz on Nov 18, 2021 6:31 AM
9
answers
0
votes
33
views
asked 7 months ago

Gremlin profile execution time not matching with Gremlin Request/Second

Hi, We are doing POC with AWS Neptune and based on the gremlin profile execution time our query takes around 10-15 ms to execute. We are using db.r5.8xlarge instance which has 32 vCPUs i.e 64 neptune threads. So our expectation when we run this query concurrently using multiple Go client we should able to get around 4200 queries/sec (1000 ms / 15 qps * 64 threads) but as per the db instance Gremlin request/second metric the number of queries processed is at 950/sec. This value is way too low from the expected values based on profile output of the query. As per the gremlin status curl output during the concurrent run of the query, I see the acceptedQueryCount is always around 150 and runningQueryCount is always 64. By which I assume there is enough job present in the FIFO queue waiting for the threads to process. But in spite of that db metric Gremlin request/sec is consistently pointing at 950/sec. Need Clarification: 1) What are the things accounted for in the gremlin profile execution time calculation? 2) What are the other factors which will affect the Gremlin request/sec metric which is not accounted for in profile execution time calculation? Data Model: R1----R1R2Edge ---> R2 ----R2R3Edge--->R3 (properties: x1, x2, x3) Query: g.withSideEffect('Neptune#typePromotion',false).V("R1:1:99").as('r1').out('R1R2Edge').out('R2R3Edge').not(and(has('x1',within('-1','tcp')),and(has('x2',lte(5432)),has('x3',gte(5432))))).select('r1').id() Thanks
2
answers
0
votes
16
views
asked a year ago

ERROR: Max frame length of 65536 has been exceeded

not a particularly large data set but I receive the following error: \[code]ERROR org.apache.tinkerpop.gremlin.driver.Handler$GremlinResponseHandler - Could not process the response io.netty.handler.codec.http.websocketx.CorruptedWebSocketFrameException: Max frame length of 65536 has been exceeded. at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.protocolViolation(WebSocket08FrameDecoder.java:426) at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.decode(WebSocket08FrameDecoder.java:286) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) Max frame length of 65536 has been exceeded.\[/code] I can query up to around 20 of them but more of them I get this error (only 31 total) The vertices are all of one type/label - query is g.V().hasLabel('DATALABEL').valueMap(true).toList() the vertices do have a good few properties including set property types of json strings
2
answers
0
votes
56
views
asked 2 years ago

Vertex Filtering Significant Performance Issues

Hello, let me share a very simple query which I am getting poor performance on. I'll share the profile and hopefully someone can explain why it is taking over 10 seconds. Additional filters after the .HasLabel() are significantly hurting performance even further (2 to 3 times worse). "gremlin":"g.V().has('__graph','123').hasLabel('MyType').count()" ******************************************************* Neptune Gremlin Profile ******************************************************* Query String ================== g.V().has('__graph','123').has(label,'MyType').count() Original Traversal ================== \[GraphStep(vertex,\[]), HasStep(\[__graph.eq(123), ~label.eq(MyType)]), CountGlobalStep] Optimized Traversal =================== Neptune steps: \[ NeptuneCountGlobalStep { JoinGroupNode { PatternNode\[(?1, <~label>, ?2=<MyType>, <~>) . project ?1 .], {estimatedCardinality=1273722, expectedTotalOutput=1273722, indexTime=0, joinTime=1131, numSearches=1, actualTotalOutput=830889} PatternNode\[(?1, <__graph>, "123", ?) . project ask .], {estimatedCardinality=2854796, indexTime=887, joinTime=8371, numSearches=830889} }, annotations={path=\[Vertex(?1):GraphStep], joinStats=true, optimizationTime=0, maxVarId=3, executionTime=10614} } ] Physical Pipeline ================= NeptuneCountGlobalStep |-- StartOp |-- JoinGroupOp |-- SpoolerOp(1000) |-- DynamicJoinOp(PatternNode\[(?1, <~label>, ?2=<MyType>, <~>) . project ?1 .], {estimatedCardinality=1273722, expectedTotalOutput=1273722}) |-- SpoolerOp(1000) |-- DynamicJoinOp(PatternNode\[(?1, <__graph>, "123", ?) . project ask .], {estimatedCardinality=2854796}) Runtime (ms) ============ Query Execution: 10614.135 Traversal Metrics ================= Step Count Traversers Time (ms) % Dur ------------------------------------------------------------------------------------------------------------- NeptuneCountGlobalStep 1 1 10614.027 100.00 >TOTAL - - 10614.027 - Predicates ========== # of predicates: 189 Results ======= Count: 1 Output: 414752 Index Operations ================ Query execution: # of statement index ops: 830890 # of unique statement index ops: 830890 Duplication ratio: 1.0 # of terms materialized: 0 Edited by: austinmalpede on Jul 7, 2020 2:02 PM Edited by: austinmalpede on Jul 7, 2020 2:03 PM Edited by: austinmalpede on Jul 8, 2020 7:55 AM
5
answers
0
votes
12
views
asked 2 years ago

How are Neptune I/O's calculated for billing purposes?

I've just started experimenting with Neptune for the past month, and my usage has been pretty minimal. I've probably only got about a dozen nodes in the database, and I've been doing some queries through the Gremlin Console and through AWS Lambda functions just to familiarize myself with how things work. I'm the only person who has access to this account. I just got my first month's bill, and it says that I've had 797,681 I/O's, which seems impossible! I'm not worried about the $0.16 charge for that this month, but I'm pretty concerned about how this might scale. I've only had one Neptune cluster running for a total of 124.266 hours this month. I'd estimate that as much as 25% of that time I wasn't interacting with the cluster at all. When I was interacting with it, however, I'd estimate I was only submitting about 100 queries in an hour at the very most, and often much less than that. Yet, at 797,681 I/O's in my billing, that works out to an average of 6,419 queries per hour! Am I missing something here? How does Amazon calculate I/O's? I had assumed that a single, multi-step query would count as a single I/O, but might it actually count as 100's of I/O's, assuming that there are multiple steps, and each of those steps may need to touch multiple quads in order to find the desired information? If that's the case, it seems like the cost has the potential to quickly get out of hand. If someone could clarify how I/O's are calculated, I'd appreciate it!
1
answers
0
votes
12
views
asked 2 years ago

Vertex Dropping Issues

Hello, I have 1 million + vertices in Neptune with the following property: ('__graph','db8db2af-8b0c-49fb-bcf1-c44cae2d2e95') I want to purge all 1 million vertices in the database. I know i'll have to drop the vertices in small chunks, otherwise the query will time out. Each chunk will be responsible for deleting 1000 vertices, which takes about 2 seconds normally. This is what the query looks like for a chunk: g.V().has('__graph','db8db2af-8b0c-49fb-bcf1-c44cae2d2e95').limit(1000).drop() I'm going to show you what the profile looks like after performing the operation twice: gremlin> g.V().has('__graph','db8db2af-8b0c-49fb-bcf1-c44cae2d2e95').limit(1000).drop().profile() ==>Traversal Metrics Step Count Traversers Time (ms) % Dur ============================================================================================================= NeptuneGraphQueryStep(Vertex) 1 1 2020.007 99.99 NeptuneDropStep 0.101 0.01 >TOTAL - - 2020.109 - ----- gremlin> g.V().has('__graph','db8db2af-8b0c-49fb-bcf1-c44cae2d2e95').limit(1000).drop().profile() ==>Traversal Metrics Step Count Traversers Time (ms) % Dur ============================================================================================================= NeptuneGraphQueryStep(Vertex) 1 1 57883.392 100.00 NeptuneDropStep 0.130 0.00 >TOTAL - - 57883.522 - Why is dropping so inconsistent? It typically should take 2000ms, however we are seeing large spikes in time like the second example. A lot of the time the query completely times out after 2 minutes. Please help, Thanks, Austin Edited by: austinmalpede on Apr 3, 2020 10:59 AM
1
answers
0
votes
15
views
asked 2 years ago
  • 1
  • 90 / page