Vishal desai’s Oracle Blog

January 30, 2019

ElasticSearch Authentication and Authorization at Indices level on AWS

Filed under: AWS, ElasticSearch, Uncategorized — vishaldesai @ 10:10 pm


Customer Requirement:

Customer XYZ is planning to host multi-tenant ElasticSearch domain to provide log analytics service to multiple clients. Customer XYZ will receive logs from different clients into their data lake and selectively push relevant logs to client specific Indices. Clients should be able to login and authenticate to Kibana portal and should only have authorization to client specific indices. There will be separate set of indices created for each client starting with standard prefix such as clientid or clientname.


In current solution, I will demonstrate how to integrate ElasticSearch domain with Cognito. Using Cognito, customer can create users for different clients in user pool. Customer can also federate using SAML provided users are available or can be created in hosted/cloud AD. Each user either from user pool or federated will map to one of the groups in Cognito and each group will be associated with IAM role which in turn will provide authorization access to set of client specific indices.

Note: Solution will work only for dev tools in kibana and not for discover, dashboards etc

Prerequisite resources:

KMS key, VPC, subnet ids, security group should be setup prior to implementation steps.

Implementation Steps:

Step 1: Elasticsearch Domain

Create ElasticSearch Domain using CloudFormation template from Appendix A.


Step 2: Sample Indices

Login to Kibana portal and create client id specific indices for testing. Kibana end point can be accessed using ec2 windows in public subnet or via proxy to bastion.

PUT orders/_doc/1

"user" : "order",

"post_date" : "2009-11-15T14:12:12",

"message" : "trying out order"


PUT customers/_doc/1


"user" : "customers",

"post_date" : "2009-11-15T14:12:12",

"message" : "trying out customer"


Step 3: Cognito

Create user pool and identity pool using CloudFormation template from Appendix A.


Step 4: Cognito Domain

Create domain for Cognito user pool.

aws cognito-idp list-user-pools --max-results 50 --output text | grep espool | awk '{print $3}'


--Domain name value in below cli must be unique

aws cognito-idp create-user-pool-domain --domain espooldemo --user-pool-id us-east-1_ldkzTlRck


Step 5: Update ElasticSearch domain with Cognito configuration.

aws es update-elasticsearch-domain-config --domain-name esdomain  --cognito-options Enabled=true,UserPoolId=us-east-1_ldkzTlRck,IdentityPoolId=us-east-1:fb6e132c-3711-4974-866f-cc4a3db7d6fa,RoleArn=arn:aws:iam::xxxxxxxx:role/service-role/CognitoAccessForAmazonES

Domain status will change to processing. Once the processing is complete status will change to Active and Cognito configuration will be updated for Elasticsearch domain.


Step 6: Users, Roles and policies

Create IAM policies, roles, Cognito user pool users, groups and map groups to IAM roles. Policy documents can be found in Appendix A. IAM policy documents are key on how to control authorization at indice level.

aws iam create-policy --policy-name client1_policy --policy-document file:///Users/xyz/Downloads/client1_policy.json

aws iam create-policy --policy-name client2_policy --policy-document file:///Users/xyz/Downloads/client2_policy.json

aws iam create-role --role-name client1_role --assume-role-policy-document file:///Users/desaivis/Downloads/client_policy_trust.json

aws iam create-role --role-name client2_role --assume-role-policy-document file:///Users/desaivis/Downloads/client_policy_trust.json

aws iam attach-role-policy --policy-arn arn:aws:iam::"xxxxxxxxxxx:policy/client1_policy --role-name client1_role

aws iam attach-role-policy --policy-arn arn:aws:iam::"xxxxxxxxxxx:policy/client2_policy --role-name client2_role

aws cognito-idp create-group --group-name client1_group --user-pool-id us-east-1_ldkzTlRck --role-arn arn:aws:iam::"xxxxxxxxxxx:role/client1_role

aws cognito-idp create-group --group-name client2_group --user-pool-id us-east-1_ldkzTlRck --role-arn arn:aws:iam::"xxxxxxxxxxx:role/client2_role

aws cognito-idp admin-create-user --user-pool-id us-east-1_ldkzTlRck --username client1_user --temporary-password Eskibana1#

aws cognito-idp admin-create-user --user-pool-id us-east-1_ldkzTlRck --username client2_user --temporary-password Eskibana2#

aws cognito-idp admin-add-user-to-group --user-pool-id us-east-1_ldkzTlRck --username client1_user --group-name client1_group

aws cognito-idp admin-add-user-to-group --user-pool-id us-east-1_ldkzTlRck --username client2_user --group-name client2_group

Step 7: Update ElasticSearch domain resource policy.

Find roleid using following commands:

aws iam get-role --role-name client1_role

aws iam get-role --role-name client2_role

Create policy document as per Appendix A.

aws cognito-identity get-identity-pool-roles --identity-pool-id us-east-1:fb6e132c-3711-4974-866f-cc4a3db7d6fa | jq '.Roles.authenticated'


aws es update-elasticsearch-domain-config --domain-name esdomain  --access-policies file:///Users/xyz/Downloads/espolicy.json

Wait for cluster status to be active.

Step 8: Testing

Login as client1_user in kibana.




Login as client2_user in kibana.



SAML integration:

Further, if customer wants to provide federation for AD users, identity provider can be configured using SAML using the xml file obtained from ADFS.


Enable the identity provider in the App client Settings.


Kibana Login page will now look like as follows:


Federated identities may not map to any groups initially so federated identities can still get access denied message. Cognito pre/post authentication triggers can be configured using lambda to add external identities to Cognito groups depending upon SAML attribute values or can be added after some verification/approval process.

Appendix A:



  1. Thank you very much for this blog entry!

    Unfortunately, it looks like the Appendix A link is broken – would it be possible to get clarification on the policy components?

    Many thanks,

    Comment by John Major — July 26, 2019 @ 6:27 pm

    • Sorry for delay John. Fixed link.

      Comment by vishaldesai — August 5, 2019 @ 8:53 pm

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at

%d bloggers like this: