Irregular activity in your AWS account - Suspicious Billing with SageMaker

0

Hi,

I have been charged for almost 2400€ for AWS SageMaker that i didn't use or had activated in my account. I use a password with a combination of Letters, Numbers and special characters and also use MFA authentication to my AWS account .

Checking the Event History without any login from my side or using my credentials i see all the following events done in my account :

Event name
Data
ListPolicies
AttachRolePolicy
CreatePolicy
CreateRole
CreateEndpoint
CreateEndpointConfig
CreateModel
DescribeRepositories
GetAuthorizationToken
CreateRepository
ListEndpoints
GetServiceQuota

On those events are the creation and activation of the SageMaker , how can it be possible to someone activate roles\services or anything on a "secure" account without the user login credentials and MFA authentication code !!!

I've followed all the steps that AWS support had sent to remove all the active services that i didn't activate, also i have a ticket open for 14 days to be refunded for the value that was charged to my card, talked several times in the support chat and the answer is that "I've checked in with the service team and there's no update as yet" ...

How can we trust and be safe if is possible to activate services on our account without our credentials and MFA authentication code ????

1 Answer
0

In the CloudTrail records there is a userIdenity element. What's the identity?

profile pictureAWS
EXPERT
kentrad
answered 2 years ago
  • Hi,

    I see on the events that the user is root, but in the event details for the userIdentity shows some strange behavior.

    I see the following , i have compared the event record for the creation of the sagemakerRole that was not done by me, and the event details with the delete of the role that was done by me, i see that the creation don't have a "sessionContext" and the mfaAuthenticated and the one that was done by me has that information.

    See the following :

    This was for the creation of the sagemakerRole

    { "eventVersion": "1.08", "userIdentity": { "type": "Root", "principalId": "<REMOVEDID>", "arn": "arn:aws:iam::<REMOVEDID>:root", "accountId": "<REMOVEDID>", "accessKeyId": "<REMOVEDKEY>" },

    This was for the delete of the sagemakerRole

    { "eventVersion": "1.08", "userIdentity": { "type": "Root", "principalId": "<REMOVEDID>", "arn": "arn:aws:iam::<REMOVEDID>:root", "accountId": "<REMOVEDID>", "accessKeyId": "<REMOVEDKEY>", "sessionContext": { "sessionIssuer": {}, "webIdFederationData": {}, "attributes": { "creationDate": "2022-08-03T08:27:56Z", "mfaAuthenticated": "true" } } },

  • Someone is using your root access keys. The MFA protects Console logins. I verified with my account that MFA is not needed using the CLI using root access keys. You should not create access keys for root unless you have to (Like setting up MFA delete on a bucket). I would disable these soonest. See: https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html

  • Hi , what was strange in my account have been deleted once i got the alert email, also confirmed with the support that, i had created my account as a free tear account for some tests and once i completed them i deleted all of the used resources. The issue started on the 07/07 someone hacked my account and changed the email , i was able to restore the account and change back to my email, have done the security recommendations at the time that was to change to a new password more complex and add the MFA authentication, once i have done that i assume the account was secure and that the MFA security was always needed to access and change anything in my account.

    What i see talking with the support was that not only my email was changed , much more in my account have been added and in regions that i never had used , also i didn't got any notifications on those changes, if it was created access keys as a security mesure if the email is changed and also the root password, the existing keys should not be valid anymore , this is a security recommendation that AWS should implement to prevent this to future users...

    What i see as a big issue is how can it be possible to do big changes on an account without using a machine or service that is under that account, my environment should be limited to itself , and what i see from the event logs is that the sourceIPAddress from those events nothing have to do with my environment or are know to me... I apologize for showing my dissatisfaction

  • You can do those sorts of restriction (source IPs) with IAM Users or Roles but not the root user. This is why it is recommended to only use the root user when you have to and for other activity use a User or Role. Also, if GuardDuty had been set up, it would have notified you of this activity. I got several GuardDuty notices running tests for this question.

  • I understand your point, but you need to see another perspective, AWS is available for all , persons with almost no IT experience or AWS knowledge to experts, if a free tear account is created logically the intension is to use the free available tools and don't have absurd bills of more than 2000k, using GuardDuty is a good idea but is not free. All those security measures and recommendations that you say make perfect sense on a company solution and needs, AWS billing should not treat free tear clients as an company and allow this absurd money charges for something that is complex and difficult to control.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions