Choosing Partition and Sort Key for the DynamoDB for a job queue


DynamoDB newbie question

I am trying to build a job table which really acts as a priority queue(One Lambda fills once a day and other lambda executes by reading one at a time). Below is the high level flow:

  • First Lambda --> reads the job parameters from JSONs and inserts to DDB table --> Second Lambda --> reads each job and executes the job accordingly.

Trying to see what Partition Key and Sort Key would make sense? Couple of things I thought of

  1. UUID as JobID: But Second Lambda wouldn't know this random ID to read from the DDB Table
  2. IndexName#RunDate: For any reason, if the first lambda re-runs after few hours, records shall be overridden and the previous state is lost.
  3. IndexName#RunDate#Run1: This needs that first lambda to read the records to check if the records are already present and then increment the number

I am looking for some simpler approach.

gefragt vor 3 Monaten116 Aufrufe
1 Antwort

You mentioned priority but I'm not sure I understand the requirements around that. Are all the jobs for a particular day processed with the same priority? Or do you want to have them processed in a preferred order? Maybe you meant that the priority is just by day (so records from the earlier date get processed first)?

I think your first thought is likely to be the right start - each job has its own unique JobID in a table that has a simple key (partition key only - no sort key). And you will likely need a GSI (or two) from there. Just need a bit more of the priority processing detail I mentioned above to give recommendations on the GSI(s).

beantwortet vor 3 Monaten
profile picture
überprüft vor 2 Monaten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen