Is profile prefix deprecated or required?

[AlainODea]

You folks say that profile prefix is deprecated:
https://github.com/aws/aws-sdk-java/blame/46c00efdd9240d2d20d734f806dcc71506af6e0f/aws-java-sdk-core/src/main/java/com/amazonaws/auth/profile/internal/BasicProfileConfigLoader.java#L96-L99

botocore (AWS Python SDK) says that the profile prefix is required:
https://github.com/boto/botocore/blob/d00ee27528efad227ad46d3db9fd2f80f802803f/botocore/configloader.py#L208-L222

Between you, it is seemingly impossible to write ~/.aws/config in a compliant way.

[millems]

This is a definite pain point. It's surprisingly difficult to create a file that makes both the CLI and Java SDK happy. This is fixed in the Java SDK 2.0, but is hard to fix in 1.11, because 1.11 uses the same code path for the ~/.aws/config (that MUST have the prefix) and the ~/.aws/credentials (that MUST NOT have the prefix).

See this issue for more discussion on incompatibilities: #803

If you're willing to work with pre-GA software, you could add a dependency on V2's auth module and adapts its ProfileCredentialsProvider to a 1.11.x credentials provider.

Otherwise we can update the warning log to not say it's deprecated (since it's not technically deprecated anymore - it was at one point which points to the chaos around these files), but we'll still need the log to say to use the 'profile' prefix when specifying the credentials provider.

[AlainODea]

@millems thank you for the clarification. I really appreciate it.

I totally understand how this can happen. Lots of fast-moving independent groups doing awesome things and then an undocumented file format sneaks in between you all.

I think updating the message to say it's not deprecated would reduce confusion.

[millems]

How is "Your profile name includes a 'profile ' prefix. This prefix is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when it is referenced from your Java code."?

[AlainODea]

That’s great wording! Thank you.

[dagnir]

We have a fix for this upstream and it should be part of the next release.

[joelittlejohn]

Hi @millems. I don't use profiles via the Java SDK (I only use them with other CLI tools). Since this warning was introduced, I now see it repeated 10 times every time I run an app or library that uses the SDK:

Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.

I understand there was a desire to make things clear, but the fix is really obtrusive. Configuring log4j in every app/library I use to turn this off isn't really an option.

Could this warning be replaced with a change in the docs somewhere? Even the comments on this GitHub issue are likely enough that Google will now provide the answer if anyone is confused.

[mibollma]

I agree with @joelittlejohn.
This annoying warning is spamming all my logs and has almost no utility.
I'm also not using profiles with the Java SDK but instead to switch between stages on the CLI.
Please either follow the suggestion of @joelittlejohn, or limit the output to occur only once e.g. through a static variable or provide a simple global way to mute the whole library.

[laymain]

I have to use both v1 and v2 SDKs in the same application with a custom profile, that is the only way I've found to make it work:

• Setting the environment variable AWS_PROFILE to custom
• Declare two sections in the ~/.aws/config file

[custom]
region=eu-west-1

[profile custom]
region=eu-west-1

But I'm having annoying contradictory warn messages from both SDKs

• SDK v1

WARN  [com.amazonaws.auth.profile.internal.BasicProfileConfigLoader] - The legacy profile format requires the 'profile ' prefix before the profile name. The latest code does not require such prefix, and will consider it as part of the profile name. Please remove the prefix if you are seeing this warning.

• SDK v2

WARN  [software.amazon.awssdk.profiles.internal.ProfileFileReader] - Ignoring profile 'custom' on line 4 because it did not start with 'profile ' and it was not 'default'.

Is there anything that can be done to avoid this?

I think there's also an issue in the BasicProfileConfigLoader: it builds a map of profile properties using the profile name as key, but it does not remove the profile prefix.
So when you have set AWS_PROFILE=custom, it does not load your profile because it stores profile custom in the map of profiles.

[millems]

@laymain Sorry that this is annoying. V2 had to be changed to be more compatible with other SDKs, which broke some compatibility with V1 itself.

For a temporary workaround, when you're using both in the same application, you could use the V2 profile credentials provider with the V1 SDK by implementing V1's AWSCredentialsProvider and delegate to the V2 DefaultCredentialsProvider?

We could see about adding a "forward compatibility" flag with the V1 SDK that allows it to behave consistently with the V2 SDK/CLI/etc.

[laymain]

@millems Thank you for your answer.

To implement your workaround, I also have to implement a wrapper for V1's AwsRegionProvider,
but V1's AwsClientBuilder abstract class does not allow its injection:

aws-sdk-java/aws-java-sdk-core/src/main/java/com/amazonaws/client/builder/AwsClientBuilder.java

Lines 69 to 95 in 1f21df5

	/**
	* {@link AwsRegionProvider} to use when no explicit region or endpointConfiguration is configured.
	* This is currently not exposed for customization by customers.
	*/
	private final AwsRegionProvider regionProvider;
	private final AdvancedConfig.Builder advancedConfig = AdvancedConfig.builder();
	private AWSCredentialsProvider credentials;
	private ClientConfiguration clientConfig;
	private RequestMetricCollector metricsCollector;
	private Region region;
	private List<RequestHandler2> requestHandlers;
	private EndpointConfiguration endpointConfiguration;
	private CsmConfigurationProvider csmConfig;
	private MonitoringListener monitoringListener;
	protected AwsClientBuilder(ClientConfigurationFactory clientConfigFactory) {
	this(clientConfigFactory, DEFAULT_REGION_PROVIDER);
	}
	@SdkTestInternalApi
	protected AwsClientBuilder(ClientConfigurationFactory clientConfigFactory,
	AwsRegionProvider regionProvider) {
	this.clientConfigFactory = clientConfigFactory;
	this.regionProvider = regionProvider;
	}

[millems]

@laymain Yeah, unfortunately region providers are an internal concept right now. Since regions don't expire like credentials, we didn't really ever plan to make them external, since configuring a region provider should be the same as configuring the region from that provider.

[laymain]

@millems For now I will stick with my ugly workaround.
But I have to disable warnings in the log4j.properties of every application that uses either SDK v1 or v2:

log4j.logger.software.amazon.awssdk.profiles.internal.ProfileFileReader=ERROR
log4j.logger.com.amazonaws.auth.profile.internal.BasicProfileConfigLoader=ERROR

It will be nice to have this "forward compatibility" flag.

[umeshksingla]

Oh god, so true

[joroKr21]

Can't you just remove this log?

[rehmanuet]

For the love of God please remove this error.

[millems]

Today's release should hopefully fix the log spam. The 1.x SDK will now fall back to "profile X" if there is no "X" profile defined and the SDK is configured to look for profile X. There is no longer a warning if a "profile X" exists in the file.

[justinTM]

still happening in org.apache.spark:spark-hadoop-cloud_2.12:3.3.0

[wtfzambo]

like @justinTM also happening in org.apache.hadoop:hadoop-aws:3.3.2.

Oddly enough, this didn't happen with org.apache.hadoop:hadoop-aws:3.2.0

[natamo]

Also happens in image for Glue: amazon/aws-glue-libs:glue_libs_3.0.0_image_01

Part of the log:

23/01/26 06:08:03 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
23/01/26 06:08:03 WARN BasicProfileConfigLoader: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.